Not disagreeing here.
why do you think a D&D clone did take the second spot the second time around?
Because 4E felt, to a lot of people, not like D&D. And the reason D&D is popular is that people like playing D&D or something like it! (That core loop of kill monsters, gain loot, and level-up is VERY strong - strong enough to be adopted by computer games, and still going strong today). Plenty of people tried 4E, but they didn't like it enough. And so they stuck with 3E - even though the publisher had changed. That's enough people to take 2nd spot, even if the numbers were far less than those that were with 3E in the day.
Vampire itself was unusual. A very strong role-playing experience. I don't know if we've really had that since - yes, there are good role-playing RPGs since then, but rarely paired with a strong theme that draws people in.
One of the problems I have with the idea that "D&D must be weak to allow other RPGs in" is that I don't think it's supported by evidence. I think games do well when they are good and when they are lucky. You need both. (Mind you, releasing a new RPG in the year of a new major edition of D&D might not be great. That would be the years 2000 and 2014).
Although, thinking about it, how much D&D is saturating the market may have an effect. With 3E, the early years saw the d20 Glut, when any other RPG would likely struggle due to stores not having enough money to stock everything available. But post-2003, I don't think that would be the case any more.
But the thing with 5E is that there hasn't been a glut of products. Nowhere near the same scale. And there have been some rather big releases of non-D&D material. Albeit, some not all that well handled. 7th Sea, for instance?