The people who were angry at RMS's "politics" were always actually mad at the concept of Free Software in general, and the fact that RMS thinks that software that is not Free is bad software (and does not think you are an exception.) He wasn't banging on about trans rights or Muslamic invasions, but people being enslaved by machines run by the powerful.
Doesn't mean that he didn't have opinions, he just knew why people cared what he thought. When he ventured a bad (but common) opinion about sleeping with teenagers, it opened a hole that his enemies drove a truck through. It's also notable that he did this to defend his friend Minsky; people often compromise their professionalism over personal relationships and interests.