I don't agree. I realize I risk being dubbed racist, but I'm really not. I have a lot of Asian friends, and a few black ones (which is mainly because the place where I live doesn't have a huge black population, and almost none in my school). I guess racism has played a part in everybody's life, true, but my home country doesn't have other races at all, so it's never been a real issue in my life.
I have never consciously thought "I only want white men", but as I've explored my sexuality, I realized I am just not physically attracted to black men, and VERY rarely attracted to Asians. There is just no physical response on my part. It's the same as with girls - I can appreciate when they are good looking, cute or hot, but it's purely analytical...
That said, I have a friend who ONLY likes Asian men. He just doesn't get attracted to other races. And I know for a fact that he IS racist. So I really don't think it's racism that defines who we're attracted to.