disclaimer: this is my opinion as a Chinese-Born US Citizen. My post is written from that perspective so I'll generally use Chinese instead of Asian as I can't speak for all Asians, but I believe most of what I say can be applied that way.
For the record, racism is a very strong accusation that implies a person's perceived superiority of a certain race or the inferiority of another race. This is why the "n word" can be considered racist, but generally I don't think the same argument can be made as strongly with the word 'Oriental.' People who use that word on others just sound somewhat archaic or ignorant to me.
There are many Chinese Americans who are fine with whatever you call them and there are some who hate being called anything but their name. The difference isn't in what's PC or "considered offensive" or whatever. The difference is in the cultural context of the person.
It's my opinion that Chinese culture does not figure in heavily into American culture. By that I mean that Chinese culture and American culture have largely remained separate entities. It is often important for a person to identify with a certain culture. For people who are originally from China or grew up in a very Chinese neighborhood, it might not be much of an issue for them to be called Chinese, or Asian, or even Oriental.
However, for the slowly growing group of Chinese/Asian Americans (like myself), we are sort of at the cross roads. My parents, although originally born in China, have done very little to bring me up in Chinese culture. Rather, they scoff at my brother and me as American-born (they use a somewhat pejorative term specifically for westernized Chinese) and regard it as an inevitability of moving to the US. This is very apparent to any westernized Chinese people who go to a Chinatown and attempt to talk to some people working there. Although I don't think my Chinese is that bad, I've been chuckled at and given the "oh children these days.." sort of response when asking for something. Sometimes they'll respond in English which is kind of annoying because I can usually understand Chinese much better than broken English. Even my own grandmother asked me why American born Chinese youths seem so dumb (she is too old to imagine a person who looks Chinese to not be able to speak Chinese well, and so she take the lack of conversation as a sign of unintelligence - this is not uncommon). This illustrates that between the cultural and often-times language barrier, Chinese Americans, and probably many Asian Americans, feel disconnected with the social circle and culture of their heritage.
On the flip side, I also don't feel that there is a very distinct Chinese American culture in the US in particular. What's there is often a preservation of Chinese culture, and, though certainly wonderful and important to me, lacks an American distinction and American context. This is slowly changing, but this was certainly the case when I was growing up. The result of this is that Chinese people are understandably seen at first-glance as outsiders in a very general way. This sometimes poses a challenge to Chinese Americans because now they may feel excluded from the very culture they feel closest (e.g. think of how a guy who acts even the slightest bit effeminate automatically adopts the whole pop-culture gay-man dossier and loses his individuality). This is the key to the whole issue. This is also what might be called the Asian American Identity Crisis. We don't very nicely fit into either our heritage's culture, or the one we grew up in.
So the moral of the story is that words like "oriental" don't offend people because the words are intrinsically offensive, but because they invoke the idea that the person referred to is an outsider, that the person referred to is so generic that no other more individualistic descriptors can be used to describe the person (obviously a police description is a different story). And though this may not necessarily seem like a big deal (like to the person who said he was called a "white person" when in an Asian country), it is different in the context of a person who doesn't associate him/herself as an outsider. And you may feel that this is still not a good enough reason to second think your words, but at least you'll understand why it's an issue to some people. I think people generally want you to know them as individuals, especially when they're on your team.
And before you say "stop being such a wimp and do something about it" - well we are, and I hope to contribute. The key to cultural change is the natural formation of the values and philosophy that distinctly make up who we are as Chinese or Asian Americans, shaped around the common human condition of individuals, not group-think and self-segregation. This change is fueled by outward manifestations of culture - art, literature, music, films, that are made by me, and maybe you, depicting our philosophy and world-view as individuals, categorized not by the color of our skin or the shape of our eyes, but by the it-just-so-happens coincidences of how people react to the same social conditions. Doesn't this sound like a perfect analogue to queer culture? Maybe it's because identity is something everyone deals with, be it through race, sex, gender, orientation, etc. when we realize there are people out there who are different from us.
No wonder some people prefer just to use Oriental! PC is getting WAAAAAAAAY out of hand - 