If you look up the anecdote, it's less the fact that he said "You're a woman!", and more the fact that that's literally all he said to her. He just kept repeating that in astonishment as she tried to talk to him, until she finally gave up.
Being surprised someone with a stereotypically male...