Elmos, I hear you.
I found a photo of myself, shirtless, when I was 23. I showed it to some people just to let them know what I looked like before I ever seriously started working out (although I did it in high school in 1966, it wasn't serious, 3-days-a-week stuff, just a high school kid fooling around, trying to be strong like Samson).
People project their own insecurities onto you, which just shows you that they don't like themselves. It's too bad they can't just like themselves, skinny, average or whatever. I never had any problem finding a guy attractive as long as I felt like I "liked" him. There was a lot of that "attitude" thing going on in the gay community in San Francisco in the 70s and 80s before working out REALLY took off. My best friend had a killer (not the sculpted, ab-6-pack type of body, but pretty muscular without the hyper-definition so many guys think is the "norm" if you want to be considered "hot") figure and when guys would meet me, they'd say, you're nice, but your friend Odis has attitude. I'd see red at that, and challenge them instantly, telling them that what they saw as "attitude" was a just a guy having nice manners and not imposing himself on others.
I used to like some pretty skinny guys, and they were astonished I found them attractive. Even my current boyfriend says he never thought he would attract someone like me (He's short, 5'6" - which is a major turn-on for me - and sort of geeky looking). I point out that I'm 62, and hardly a shade of what I used to look like (still have a big chest, decent shoulders, but tore my long bicep on my left arm last September and have rotator cuff issues, so unless I want shoulder surgery, I take it VERY easy), and still have nice arms, which I can pump up very fast, except the left one looks like - LOL - "Popeye" bicep (the muscle is further down, closer to the elbow than up near the middle of the bicep).
Anyway, it's great that guys want to be 'built,' but the underlying reason for it is more, these days, like a cockfight. It used to be you had to have a big dick. Now, you can overwhelm other guys just by being hyper-muscular (and if you weren't gifted with a big dick, for some guys, it takes precedence over having a big dick). But watching the younger generation guys work out? It's the equivalent of women hating someone with a figure like Raquel Welch had in the 60s. You'd think people would have enough self-awareness to realize that beauty comes in ALL shapes and sizes. If I saw a cute 5'7" guy standing next to someone built like the guy in the 2nd photo, I'd be shoving that guy aside to get the the little guy, smiling at the little guy to let him know I was WAY interested in him.
I wonder if it's the hype behind having muscles that makes so many guys want someone muscular, like having the "trophy-boyfriend" who's built like the proverbial brick shithouse. Do they even look at the person's character (his virtues) or is it all superficial beauty stuff that makes us lust after the big guys. I didn't like the attention I got in SF in the 70s: it was all sex, and besides, the really nice guys wouldn't come near me, so I'd have to sidle over to them and comment on something they were wearing to get the ball rolling. Once I did, they warmed up.
I can't blame guys for finding muscular guys sexually desirable, but it's not the only perspective out there. Even my buddy,who died from AIDS, God bless him (he really was sweet-natured) the really buff one? He went for the same kind of guys I did: shorter and especially if they were shy, he'd go ape-shit over them: he found that so appealing.
So, you guys who aren't 'built'? Don't automatically assume that every guy wants you to look like the actor who played Thor. "Average" guys are just as hot as muscular ones, and frequently because, paraphrasing a line from The Wizard of Oz : they've got a heart.