I think you missed the quick discussion of Karl Popper's paradox of tolerance we had a while back. There's been an answer to this since at least 1945. Here goes again, because this 'why aren't you tolerant of my intolerance?' whining comes up literally. every. time.
"if a society is tolerant without limit, their ability to be tolerant will eventually be seized or destroyed by the intolerant. Popper came to the seemingly paradoxical conclusion that in order to maintain a tolerant society, the society must be intolerant of intolerance."
https://en.wikipedia.org/wiki/Paradox_of_tolerance
"Nerd" and "basement-dweller" aren't the problematic parts. It's the "misogynist" part.