I know a lot of people who feel this way and it always confuses me. If it's a good song, it's a good song... no matter who else knows or likes it. I can understand that sometimes songs get over-played, but to actively not like a song because it's popular (or not, as is often the case on this board) is odd to me.
I know a lot of people share your opinion as well. Take, for example, an artist such as Vanessa Carlton. Most people know her by her poppy single in 2002 "A Thousand Miles" (and maybe even her second, "Ordinary Day"). But basically after that, her songs dropped off the radar. She had a second album, released in 2004, but because her record label did a bad job of promoting it, nothing really happened, and nobody really caught on.
Vanessa has a lot better, deeper, meaningful songs than her only hit that most people know her by, but because some people only listen to radio/mainstream music, that other side doesn't get exposed, and people just classify her as one of those Avril Lavgine-y, Michelle Branch-ish kinda girls, when she's not.
People get the wrong sense of who she is, as well as her music.
In the case of Rufus, however, I think "Going to a Town" reflects his style of music well and although he isn't wildly mainstream, it works for who he is as an artist.
There's probably more I could say, but I'll leave it at that.
Oh, and by the way, Vanessa's first single "Nolita Fairytale", off of her coming CD, is out! Have a listen!