I'm going back to square one here to address the original poster's question to start with. And that question was, to paraphrase, "Am I the weird one?"
Well, I think it turns out you're the minority, and if that's your definition of "weird", then you are. But I think you're asking if you're in the wrong, and from my point of view, there's not enough info supplied to say. Whether or not it's "weird" to want to play cross-gendered PCs or to want to ban them depends on why you want it that way. And if I'm not mistaken, you still don't know why you feel the way you do.
I do play cross-gendered PCs. I can tell you why I do it, to. I concieve my character concepts the same way I would concieve them if I were writing a novel or a video script. As such, they come out all kinds of ways, including male and female. If I've concieved the character one way and I like it, and it turns out to be innappropriate to the game at hand,I would rather save the idea for another game and make a new one for the current game. If that means honoring the DMs request that I play male characters only, then my femmes can see action elsewhere, no biggie. Even if it's just the pronoun issue, I've seen it come up and can understand that some people are really annoyed by it. I've seen some things described on this thread, though, that I can say I'm glad I've never seen. I can understand the "risk management" arguement that some have put forward as a very valid reason for the ruling based on what's been hinted at Teflon Billy having to endure. So TB would ban half of my characters ... and it doesn't make either of us weird. He doesn't want to be reminded of some people who
were weird, and who would have weirded me out, too. Fully understandble. I have the same hang-up with what I call "freak show D&D"; the party is made of a drow, a minotaur, a half-dragon (gold), a dwarf with the infernal template, and an awakened carrion crawler -- or some other hodge-podge of races anyone of which would be unusual and make no sense together. It could be done well, but I've seen some bad ones and heard stories of worse, and I just don't want to have to deal with it.
But I digress.
So yeah, the group should decide what works for 'em and what doesn't. Then do things that way. I know, every issue in the game always seems to come down to that. In this case, though, the original poster seems still not sure about his decision because he's not sure where it comes from. So I think the way we can all be constructive is to explain the origins of our decisions, not bicker and name call each other over them.
Again, I create characters like I would for any fiction, and to say that a male can't create a convincing female character or vice versa seems like saying that characters in novels and movies should be limited to one sex only. That's kind of limiting, I think. [Edit: and so does Gez, apparently, who posted a similar remark while I was typing this up.] If you say that it only applies to the main POV characters, you're still trying to take away my Honor Harrington books.

The arguement about cognitive dissonance over visuals I can see, but I also think that in many cases, it can be overcome with visual aids -- if one of your group can draw, or you can find artwork that fits your image of your character, you can say "This is what Vilasonee looks like, even though I look nothing like it, think of this when "she" speaks." And that never hurts in a game where you do have some physical resemblance to your character, for that matter.