I'm a 70s Gen Xer and my interest in fantasy stemmed from my mother, who read it (and scifi). I started by drawing creatures and reading books like Lloyd Alexander's Prydain series, Ursula K Le Guin, and Tolkien, and even my first attempt at world-building sometime around then. When I was gifted a set of the AD&D hardcovers around 1981-82 from family friends who lost interest for video games, my interest supernova-ed.
But I was never a "nerd" - not in the visible, stereotypical way that was common back then, that you see represented in films from the era. But that was also true of quite a lot of D&D players during the 80s boom. I remember bringing
Deities & Demigods to middle school in the early 80s and pouring over it with friends, and never being picked on for it (though I was at a Buddhist private school, so maybe things would have been different at public school).
I think D&D contracted back in the late 80s from mainstream to the nerd ghetto, and remained so through the 90s, though by that point there were "geekdom alliances" formed through crossover interests (e.g. World of Darkness, LARPing, Goth, and theater). My own personal crossovers were fantastical literature, art, and worldbuilding, with RPGs not being a singular interest, but one venue to explore and create fantastical worlds.
As is well-known, there were several cultural streams that de-stigmatized geekdom, especially starting with Harry Potter (1997), which inspired a golden age of middle reader and teen fantasy, then the Lord of the Rings movies (2001-03), Marvel Cinematic Universe (200
, and Game of Thrones HBO series (2011) - as well as several other things in-between. We can add in Stranger Things (2016) as probably being the single biggest inspiration for D&D's current popularity.
Meaning, there was a huge sea change over about 20 years starting in the mid-to-late 90s. But the sense of D&D being "shameful" probably stems back to the ghetto-ization of nerds and geeks, and the general tribalism of middle and high school in the 70s and 80s. It has lingered on in various ways, but is far more muted now than even 20 years ago...at least as far as I can tell.