Yes, well we're adults not toddlers. Sometimes you have to think of something other than your pecker. Or your feelings.
This situation is going to destroy this woman. Either through a life of quiet desperation, or dramatic discovery. he's stealing, her dreams, her self respect, her time, her security, her chance at happily ever after. When she finds out - and they always do, he'll have pushed her into that special hell reserved for wives of closet cases.
Do you think she'll thank him for that? Do you want to be a part of that? How can you ignore your role in it?
It's time to be a man, to be a stand up guy, you can't stop him from hurting her - the sad thing is the damage is already done - but you can walk away from hurting her any more than you already have.
Walking here is not only the right thing to do, but it has the added benefit of being the prudent thing to do, because if you think being the lesser piece of ass isn't going to affect your self respect, and self esteem; you're lying to yourself just as much as he's lying to her.
Yes it's hard, and yes I'm being hard, because there is third party abuse going on here. You are hurting someone else, for no better reason than your pecker and your "feelings."
You have no right to insert yourself into her marriage. Even if he invited you in, you are responsible for making that choice.
It's time to fix it, time to walk away. Time to work on yourself, and the kind of person you are.