It seems to me he is very comfortable...
NO! not with you or in your presence, that may be, but his comfort is based in his familiarity with uncertainty! Uncertainty and unease became a game, the game routine, the routine mundane, the mundane his comfort.
It seems to me he uses the "joking" as an excuse to touch you, when he does he laughs it off as a joke. Look deep enough and peel away the layers of this cat-and-mouse-like game and you will find this uncertainty at the core.
I put forth that he indeed does want to touch you, not only as he has done but intimately. Unfortunately his actions are connected fiercely to uncertainty. A lack of certainty you have... contributed to. By taking no action and being complacent in his 'jokes' you have reinforced his course of action and assesment as being true.
Perhaps he gets up the nerve to touch you, only ot laugh it off out of fear that should you know the true reasons you would go 'hetero-ape-shit' on him.
The game is a cycle. He jokes with you in a sexualized way, alluding to his true intent, laughing it off with caution. You reinforce his actions by giving no clear signals.
When you attempt to return his sexualized actions he recoils, perhaps maybe it is only a game and he is really only having fun. Perhaps he wants, desires, needs this attention but recoils out of fear. A fear born of distrust derived from your game. Could he handle it, would he be strong enough to only "joke" with you.
The cyclical patterns will continue.
That is until the cyclical behavior ends. Throw the track, hit the brakes, hold up a sign, something.
Next time he does something sexually charged, touches you, grabs you, whatever make it clear you would not mind.
Perhaps the next time you turn around you don't say anything, just look him in the eyes. Silence would reign supreme between you two for a moment, allow him to break the moment. probably "what?"
Assertive. Leave no wiggle room, doubt, uncertainty. Leave no room for interpretation, uncertainty has allowed the game to go on. You not coming out, him not saying he wants you, you playing his game. "When you touch me, I don't mind." Don't laugh it off, you would only reveal a new tool in the game, joking about being serious.
His reaction would be less predictable. perhaps he gets out of bed, maybe he asks "you don't?", maybe he reasserts he is only joking, he might possibly move his hand slowly to your face. Whatever his action reassert yourself, make it known that you have no qualms with his actions.
He has been told now you don't have a problem with his actions, you are not averse to them. Take another step based on his reaction, if he's heading out the door carrying his shoes... let him be. If his hand is across your cheek or neck then tell him you want him to touch you. "I like it."
The key though to ending your conundrum is ending the cycle of behavior. Pull away the layers, get to the core.