There was also the socially-constructed "ideal" human that eugenics strived for, or how it designated non-white races as "inferior" genetically, or the subjugation of the individual's fitness to society's "needs". My feeling is that these ideas are "not even wrong" enough that they'll incrementally come right back if you don't place a moral firewall around the entire concept.

Sure there is, as you've just demonstrated. Compulsory sterilization is one of many ways to implement eugenics, and the fact that it is so closely associated with the general concept is the cultural meme TeMPOraL was referring to. Other more benign forms of eugenics like incest taboo or selecting your sexual partners are not just acceptable, they are often not even seen as eugenics, because eugenics is "that bad thing the Nazis did".