Your sex determines everything from the shape of your face to the size of your hands to what kind of genitals you have, but apparently some people believe that they have nothing to do with the construction of your brain. I find it very difficult to understand this point of view given the anecdotal evidence which is everywhere around us in our everyday lives, and guess it's explained away by several decades of anti-discrimination culture.
The scarey Catch 22 that people fear is that to admit there's a difference opens the door for discrimination.
For instance, african american babies were found to have better motor skills than caucasian babies. This is okay to say from a politically correct perspective for purely cultural reasons, and there's arguably strong anecdotal evidence to support it (dominance of pro basketball by african american players). You'll still find people trying to deny such things, because it's not "equal". But...do we still find caucasian pro basketball players? Yup, and it doesn't stop them from trying. Not so scarey now, is it?