I've read the thread. A lot of interesting discussion.
I agree that the current incarnation of the rules is not particularly sexist, and that the current artwork and marketing of D&D, while perhaps not perfect, is certainly worlds better than it has been in the past.
But why does the "boys' club" aspect of D&D persist? Why still such a gender imbalance among players - 80/20 in Rouse's best case scenario?
When I think about this, I have a hard time disentangling general gender relations in society at large, including instutionalized sexism, and the specific reputation of D&D and its impact on small group dynamics of gamers and how encounters between gamers and "outsiders" play out. This has been touched on tangentially in this thread but not really drawn out, I don't think.
For a number of reasons, not least the sexist 70's-era tropes from the roots of the game, D&D got tagged in the mainstream as an activity for geeky guys. That may be starting to shift a bit, but it's inevitably going to lag behind whatever changes have taken place in the rules and marketing. Thinking about potential new, young players, I think that just raises a big barrier to entry for girls and young women.
In a nutshell, to the extent that D&D is perceived as "uncool", those numberless teenage hordes who above all else desperately want to be cool will avoid it. There are of course just as many boys as well as girls who fall into this category, but add in the preconception that D&D is a boy's club, male pursuit, and I think the repellent effect will be stronger for girls.
Then, back to the young males who are actually playing. When they encounter an "eww, you play D&D, how stupid!" reaction from a "cool" peer girl, it's going to make them feel defensive. They may expect all girls will have the same attitude and seek to exclude them to protect themselves from further hurt feelings. Thus, the "boy's club" perception may be pushed further towards reality. It's a pernicious dynamic, a self-fulfilling prophecy.
These are of course generalizations, not universal truths, but they do conform with some of my experiences. Again I'm talking in particular about younger players here - I think as players get older and generally less insecure, this dynamic eases. Anecdotally, women seem to have greater representation as the demographic gets older.
Finally, that sort of "you're lame" - "no, you're lame!" in-group/out-group dynamic forming around D&D does not always or exclusively break along gender lines. Not by any means. I just have a feeling that it's common for it to play out that way. Who knows, assuming that I'm not totally off base here, this might well be reflecting deeper sexist dynamics of the larger culture in the end.