The reality is that the swastika and the Confederate flag mean different things to different people. No amount of harping on just one meaning will change that. It has nothing to do with palying with words, and everything to do with recognizing that the world is not the simple, tidy place we might like it to be.
Speaking of words... the comparison between a visual symbol and a sound would be the correct one, and then you're absolutely right: a given sound pattern has no intrinsic meaning. That can be demonstrated just using English: we have "row" and "roe", "chased" and "chaste", "picked" and "Pict", "bite" and "byte", "red" and "read", "reed" and "read and "rede", "style" and "stile", "hoe" and "ho", "aloud" and "allowed", "noes" and "nose" and "knows", and the infamous "they're", "their", and "there". And if we go between languages, the examples are even more abundant -- for example, there's a vocal sound that in one language means "frozen waste", in another means "bold spirit", and in another "avenging demon".
The same is true of visual symbols.