I have kind of a weird experience with this.
I played D&D fairly openly in high school and early college years. After taking a break for a couple years, fearing the stigma, I told myself I was "growing up" and began to conform to what society esteems as successful. For 10 years I avoided D&D and TTRPGs, worked out, raced cars, started getting into sports, took up hip-hop dancing, skydiving, and hiking in Moab and Arches national park, shopped exclusively at high-end stores, lived in the trendiest apartments, hosted parties, dressed to the nines, owned a BMW M3, etc. I did it all to get the approval of society, women, etc. I was still a "nerd" but I kept it within "socially acceptable" limits, things like Call of Duty, Lord of the Rings, etc.
It was all so fake. Most of the time I was miserable. Dating was awful because I ended up with "gorgeous" women that I had nothing in common with, that I couldn't relate to, really. This would be a turnoff to both of us.
About 6 months ago I realized that I really wasn't happy. I wound up in therapy. After two months, I confided in my therapist that I had always enjoyed writing fantasy-type stuff but wrote it off as being childish. He encouraged me to rivisit and engage that part of my life. The last time I could remember really being excited and happy was when I was playing D&D with my fellow nerds. I reached out to my old friends that I hadn't talked to in ten years to see if anyone was interested. We got a 5E campaign going and at first I tried to hide it, but ultimately just decided to own it. That was probably the best decision I ever made.
I have a smaller, but much closer group of friends now, the women I date know that I D&D right off the bat. It's my pre-qualifier, if that's a problem for them, I know that it's not gonna work for a variety of reasons. Most of them wind up being curious, rather than repulsed. A lot of this has to do with leading a balanced lifestyle, I think. People write off D&D players because in their mind, so many of them fit a stereotype. I don't match that stereotype and I think that's how I've managed to open people's minds about it.
I've converted a number of "haters" to join my group since then. People that would do the nasally voice shut up once they came over to my place and saw me laboring over a detailed castle map for Hoard of the Dragon Queen's Greenest keep. Another guy came over and saw my projector table setup, minis, alea tool magnets, etc. His words were "Wow. I mean... I'll make fun of D&D as much as the next guy, but you make this look awesome." Having tangible stuff besides books to show them, having fruits of your creative efforts (maps, etc) on hand really helps to take the wind out of their sails in some cases.
Eventually, I broke the news to my parents and... they were thrilled. They're authors and they loved seeing how creative I get when I'm running a campaign. I literally get to bounce ideas off of them for campaigns whenever I have a family dinner. It's awesome, and now there's a bullywug love triangle in the mere of dead men that my players are highly amused by, thanks to my mom...
My mood changed almost overnight. I was happier. Eventually all aspects of my life (social, professional, spiritual) lifted.
Over time, I found there were a surprising number of people that played D&D that I never would have guessed. My landlord came by once, saw my stuff, and talked my ear off about his campaign that he and his buddies have been playing for years. A cute girl at my complex heard about it and informed me she plays as well, along with a bunch of others. It was surprising to me.
My group now consists of about 6 players, half are guys in their early thirties, the other half are conventionally attractive women in their early 20's. It's... not at all what I expected when I started DMing again, but I won't complain. I think nerd culture is becoming much more acceptable as a whole.
Observations
1. Use your best judgment before talking to people about it, but don't hide it (unless you wanna, but it's a lot of work, at least for me).
2. Some people are insecure jerks that need to read How to Win Friends and Influence People for basics on social interaction. Your Doctor friend is one of them. Unfortunately, there are others who are quite involved in the D&D hobby as well...
3. People have all kinds of crazy hobbies. In my case, I'm fortunate enough to have a generous salary to blow on D&D and can devote a lot of time to making my sessions awesome, people see the amount of work that goes into it and often respect that.
4. If you lead a balanced life outside of D&D, people will be much less likely to judge you negatively for it. Kick butt at work, be friendly with people, practice good hygiene, etc.
5. Pick your players carefully, if you can, from others that have balanced lives, good hygiene, and will provide a good experience for newcomers. I get calls at least twice a week from players asking what they can do to help make this or that newbie feel more welcome and comfortable. It helps a lot. Each person you convert to D&D becomes an evangelist, especially if you and your players own it and don't try and hide it.
6. Talk it up like it's the greatest thing in the world, because it is.
7. Refer to the CS Lewis quote above if people give you crap about needing to grow up. In my case, it was an article on Cracked that made a similar point that convinced me "acting like an adult by not doing things you enjoy" was stupid.
8. If you have younger family members, nieces, nephews, that sort of thing, run a (simplified) game for them when there's a family get-together. Your more curious relatives can overhear what's going on and get a feel for it, and they will love you for distracting their younger, more disruptive children. Your nephews will think you're the coolest uncle ever. It's a win-win-win. Minis and a map help with this. Let them win though, no TPKs, lol.
9. Or you can try to hide it from everyone. Honestly, that works for some. But don't be ashamed. My group's motto is "There's no shame in our game", and I mean it. I've lived the 'good life' and I got bored of it. My imagination is far more stimulating, and I'm finally around people that get me and are interested in the same stuff I am.
tl;dr - D&D is dope. Make it cool, don't be defensive, and don't be the stereotypical socially-retarded, poor hygiene, annoying, awkward know-it-all guy that people hate regardless of whether they play D&D or not. Make it your passion and people will tend to respect it. If they don't, they're the ones with the problem, not you. Don't 'sell out' to try and fit in with society's expectations like I did, you'll end up miserable.