As has crzyrazn with his "concrete evidence" clause. Prejudice isn't always easy to prove. 50 years from now when discrimination against gays is illegal, it will still happen in discreet manners that have no "concrete evidence." When it's illegal to fire gays cuz they're gay, a distraction reason will be provided.
No good can come from taking a concrete position and sticking to it, no matter what, refusing to budge or accept variables.
For that matter, I agree with the OP on one hand, but sometimes the problem isn't with the Jew or the black, rather with the person who assumes that they must censor themselves. I'm a light-hearted, easy-going guy but for some reason people feel the need to offer huge disclaimers whenever they say a sentence with the word "black" in it. "I'm not racist. I don't mean any offense. Don't take this the wrong way but....." It's ridiculous and unnecessary, the assumption is that if they say something to upset me I'm going to go on a rampage. I've seen white people tell "white people" jokes to their white friends and there's never any disclaimers or assumptions that the white person will be uber-sensitive about it.
Perhaps the problem is that you censor yourself because you make assumptions about how a Jewish or black person would respond, you assume they're ultra-sensitive and enjoy playing the victim role. Depending on the situation you could either be right, or you could be acting out a prejudice.