You can justify any kind of discrimination that way. I can come up with a number of reasons that would make women, people of color, foreigners or literally any other group a "poor culture fit".
> Sure, we probably miss some good people, but we have a specific culture in place that we all enjoy, and I don't want to feel uncomfortable or have to walk on egg shells around my team.
So you'd reject a candidate because of comfort and enjoyment? Even though the candidate could make you tons of money? That's very discouraging.
> You may just be rude in an interview, and that will 100% result in me turning you down. It doesn't matter much to me if you're an "Aspie/ADD"
There's a difference between having Asperger's and being an asshole. A person with Asperger's communicates differently than a person without Asperger's. You chose to interpret that difference as rudeness. Instead of understanding the cause of the differences and adapting to it, you simply eliminated the candidate for the sake of your own comfort.
This doesn't matter to you but it should. Especially if you want to claim you don't discriminate based on factors the candidate can't change.