I agree with trawler69 on this one ... allow him to do what he wants and see how far he goes. You have not indicated whether or not you are out of the closet. If you are not, he is trying to find out if you are also gay so he can play.
At first, I thought that maybe he might be playing with your head on purpose being in cahoots with the other friend (the footsie part) in order to see your reaction to male contact and see if you're gay. Str8 guys sometimes do that - if that was the case, your reaction probably got them to think that it was not the case. But then you said this hot guy and you were alone and he was even more frisky.

Again, let him touch you and see what he does. No str8 man is gonna fondle your cock for any length of time out of curiosity!
Reminds me of a story!
I remember about 15 years ago while taking a summer session in college I met this guy in accounting who I became friends with. I was getting signals from him that he was gay but since I was a "male virgin" I was scared to even acknowledge that it was happening. Besides, he was engaged at the time and safely assumed he was straight. One night, he called me and said that he was bored and can I come over? It was like 11pm and he lived in Queens while I was in the Bronx. It was a hot night and I was horny as hell and I said no, it's late. He kept insisting and I finally gave in (begrudgingly to him, secretly thrilled). When I got there, he opened the door and he was shirtless and only wearing a pair of shorts

... the most beautiful hairy chest I ever saw!
Long story short, we hung out and ended up watching TV on his bed. Somehow we ended up wrestling on the bed in the dark in the glow of the TV set. During the wrestling I desperately tried to hide my raging hard-on and I believe he did too. Nothing ever happened that night

but the opportunity was oh so there!
Fast forward about 3 months. I had returned to my home college and dorm. We were on the phone joking around and at one point he just came out and said, "Let me ask you something. Are you straight?" I was shocked since no one ever asked me that until then - and based on the tone of his voice and the related conversation I
KNEW what he was getting at. There was a pregnant pause during which I uttered uuuuh, "Are you?" ... "I asked you first" he says, then was like no and he admitted that he was not either - never been with another guy but that he was on the down-low especially with his fiancee. At that point I was kicking myself for not letting anything ever happen that night in Queens!

He even recalled that night we wrestled and said how terrified he was to try anything with me, and how he would have loved it to have happened
We lost touch for several months due to finals and the holidays and just plain life - during which time he married. We spoke again in May of the following year close to my graduation, when he tells me that he and his wife are getting divorced. He finally acknowledged that he was attracted more to guys and that he was dating a guy

... of course, I was jealous but at the same time kinda happy that he came out. We remained in close contact after that -- and it was probably a year after that when we finally "hooked up" ... his was the first
Real Cock I ever tasted in my mouth and what a NICE uncut cock it was!

We fooled around off and on for several years but it never got to anal sex. He has had the same boyfriend off and on all this time -- I know not cool

as I ended up being the "mistress" sort of but I did not know better at the time.
We have since lost touch, my doing due to something non-sexual and non-gay that he said that made me feel really uncomfortable as a friend - I told him then I thought it would be best if we went our separate ways - convenient for me as well since I was moving away soon thereafter. That was only 5 months ago. Ironically, he was interested in now taking things more seriously with me. It hurt to cut all ties, especially since I always had a thing for him even though I had my own relationships. I do miss him but I am practically convinced that this is irreperable even though I love the guy as a friend. But in the end, it is best as I have happily moved on and have a great new relationship I am eager to nurture further.
ANYWAY - so sorry I went on my little

... but your situation sounds a lot like how mine started and I figured you and everyone else would be entertained by a litte story
Give that man a chance! Don't regret like I did in Queens ... You both might have some fun and give each other what you both are looking for.