No, avoiding anything potentially negative is not what I'm saying. Your argument (that context always matters) leaves discourse and society highly susceptible to dog-whistles[1], by forcing all good-faith participants to interpret all communication in the most generous way possible. Bad-faith participants, on the other hand, are free to exploit that generosity.
By calling out and avoiding dog-whistles, even including accidental Nazi slogans (once pointed out), we reduce the impact of this attack on good-faith discussion and actual increase the level of openness and being up-front with our opinions.
One key difference between this and virtue signaling or thought policing is that it's the specific wording that is avoided, and not the underlying thoughts or opinions.
[1]: https://en.wikipedia.org/wiki/Dog_whistle_(politics)