So what, then, do we make of the short historical period when D&D wasn't the undisputed king of the RPG market---the mid/late 90s White Wolf incursion?
Was it actually the case? Were more people actually entering the hobby through Vampire: The Masquerade than through 1e/2e? I'll admit, I was completely out of the RPG scene around this time, other than playing Baldur's Gate for PC, so I'm asking for outside perspective.
In the late 90's, D&D probably slipped to third. My impression of the RPG scene at the time was not only was Storyteller system far and away the more popular game bringing the most players into gaming, but Deadlands also had more tables going than D&D during that period.
As for the success of VtM, I think it's best to tell a story to explain that. So, sometime during my college career I went to the book store and made a b line to the RPG books (as usual). To my surprise, someone was already there. There were two high school girls, cute ones, sitting on the floor in front of the RPG books. This wasn't anything I'd ever seen before. So I pulled some book off the shelf to browse and eaves dropped from a little ways away. They didn't notice me. They had in front of them on the floor 'LA by Night', and they had it open to a page on one of the NPCs, and the conversation I overheard went something like this:
Girl #1: "He sleeps on top of elevators at night."
Girl #2: "OMG He's so sexy"
Girl #1: "Yeah, I know. When I get in an elevator I imagine he's sleeping over my head."
Girl #2: "So hot."
Girl #1: "Every time I think about him, I think some going to swoon."
Girl #2: "tee hee hee"
Girl #1: "tee hee hee"
And so it went. For me, that was a completely new experience of what it meant to be a gamer and what might attract someone into gaming. So was VtM bigger than D&D during most of the 90's? I'd say, "Yes, absolutely." Certainly it was tapping into markets that D&D had never really touched.
Story teller is an absolutely cruddy system in retrospect, and VtM was the first game I encountered that played nothing like it was described (though it would be years before I actually played it). The game it described was about desperately clinging on to your humanity and fighting back the darkness before you succumbed completely. But because of the rewards structure and the fact that that introspective game in the examples of play could really only be played as a 1 on 1, the game as it was really played was half black wearing caped super-heroes, and half high school clique politics.
So what was going on? Was it really just an odd confluence of cultural incidences that came together at just the right moment and White Wolf capitalized on it?
Yes, and no. The timing definitely had something to do with it, but it also was definitely the first RPG that deliberately set out to be sexy and edgy and actually tapped into a market significantly larger than the nerd crowd. I found the book horrifying at some level because it felt like it was glorifying monsters, but in actual play it was usually more silly than monstrous (there were a few exceptions). Ultimately it killed itself because there is only so much you can sell yourself on shock value before you end up as schlock. It kept trying to one up itself, and it didn't help that half the production staff was stoned more than half the time.