I'd enjoy the flirtation and keep it at that if I were you. This thing could go wrong in so many ways--with you getting hurt in every one.
Suppose he's just teasing; if you show real interest, it puts him in an uncomfortable place and he's likely to get angry or resentful, or lose his respect for you. Things between you will get tense, possibly hostile, which will make your job suck big time. (It will also kill your chances to move up the ladder.)
Suppose he means it. That's even worse. There's no telling how he'll behave after you fuck around with him. But it will certainly change him and things at work.
If he really gets into it, he'll keep wanting more and because you two are together for hours on end, it will turn into an obligation that offers no practical or emotional benefits to you. He'll stay with his wife to maintain appearances; you'll become "the other" and have to be happy to get him when you can, rather than when you want or need him.
If it's a fantasy driving his comments, he'll do the deed but won't be ready to cope with the emotional conflicts it triggers. Maybe guilt, maybe confusion, maybe disappointment (no matter how great you are in the sack), maybe truth about himself--he'll feel something that he's not expected, though. And it will cloud his view of you and your professionalism.
If it turns out great, you'll be squeezed in a box, unable to talk about it at work. That will make you crazy. If not so great, you two will have to pretend it never happened, which will probably lead to distance and anxiety for both of you.
You'll know things about him that will change how you see him, too. How he looks naked, how he behaves in bed, etc. But he's not a trick that you can avoid. He's your boss.
And what if it turns out to be one-sided with one of you thinking it was amazing and the other not so impressed? How do you work around that?
If you like your job, the office atmosphere, your coworkers, etc., you can pretty much kiss all that goodbye... sooner or later.
On the other hand, if you let him flirt all he wants, a strangely exciting bond will happen. He'll realize you're respecting his position and his marriage; you'll get to enjoy attention from what sounds like a damn sexy guy.
Years ago, I was in your shoes. I had this hot, hot married boss--looked like a Kennedy, but with the body of a linebacker--who sent every imaginable signal my way. He'd park in my office for hours, closing the door as he came in and had no business reason for being there. He'd make two-sided jokes, ask me about my personal life, complain about his marriage being at a standstil. He'd rock back in his chair, legs wide open, and watch closely to see if I checked out his package. (I did, of course, and it was obvious he had feast tucked away in his suit pants.) I played along, flirted right back at him, but never indicated I wanted it go further than our little game.
I wanted this man so bad and I probably fell in love with him, too. That's how I realized that if I had him, I'd lose him somehow--and my job would either go sour or go away in the aftermath. As he moved up, he took me with him until I got a great offer from a competitive firm. When that happened, he wrote my letter of recommendation.
