In my opinion, and this is just my two cents, a lot of the D&D stigma is well-earned. In any hobby, the extremes become the stereotype. If you think of model train afficionados, the mental image you get is inevitably the 40-something man with the thick glasses, button-up shirt, and high-waisted slacks in his basement assembling his HO-scale track. When you think of oenophiles (wine drinkers), instinctively the snooty, upper-crust social elite pops into your head. And when people think "D&D", the image they get is the social outcast who lives in a fantasy world without a clue as to how to relate to normal people.
Now, we all know as gamers that the stereotype doesn't reflect ALL of us - but we all know at least one person it does apply to, and as with any hobby, that's the most visible element to those who aren't involved in it. I'll second the person that said basically if the people you associate with aren't interested in looking past the stereotype, then maybe they're not the people you want to associate with.
For me, at least, hobbies are either A) how I've met most of my friends, or B) something that comes up in conversation after I've known people for a while. For example, I came back from my local bookstore to work after picking up Eberron at lunch. Later that afternoon, one of my co-workers comes by and asks "Hey, was that a D&D book you were reading earlier? Never figured you for the type. So it's not just for nerds anymore?"
If people don't have a defined opinion of who YOU are as a person, it's easy for them to define you by your hobbies. Heck, depending on which of my past occupations/hobbies people find out about first, I often get stereotyped as a musclebound moron or an overintellectual geek. In reality, I'm somewhere in the middle. Once folks get to know me, the fact that I play D&D, read comic books, watch (and occasionally perform) professional wrestling, attend the symphony, enjoy fine dining, and write fiction - none of it affects how they think of me once they know me as something more than just my hobbies.