Hurm, lets see I was born in '80. I've not read most of what is considered the classic fantasy library, so I know that feeling of being lost when the conversation turns to a lot of stories that seem to come up in relation to D&D.
When I was little my bedtime stories were King Arthur, Robin Hood, and The Hobbit. The first thing I remember reading on my own were the Elfquest graphic novels. I read LotR's in elementary school (heh, it took me a while, started in the early in 5th grade and ended in the 6th). And I loved Orson Scott Card and Chronicle of Narnia, but beyond that not really much in the book department. Movies had a big influence like Labyrinth, Dark Crystal, Dragonslayer, and especially Willow, and of course stuff like Star Wars and Indiana Jones. These days I'm mostly into anime and manga. Stuff like Full Metal Alchemist, Witch Hunter Robin, Princess Mononoke, Spirited Away, Howl's Moving Castle, Berserk, Lone Wolf and Cub, and anything by Junji Ito (horror manga-ka) and even Fairy Tail and One Piece inform a lot of how I view and think of fantasy and gaming. Most recently, coming back around to books I've come to enjoy China Miéville.
I doubt I'm alone in that kind of experience. Really I think you're right, there is a disconnect in gamer culture, like we're cooking with totally different ingredients.