I think the reasons for D&D selling alot are:
1) It was there first, thus it is a standard-setter AND has the longest time to get media and public exposure.
2) Its base assumption (about how the game-world works) are generic enough to appeal to a very wide variety of fantasy fans, and fantasy is quite a popular genre (Harry Potter, Hercules/Xena, LotR, ALOT of computer/console games). Vampire: The Mascarade, for example, appeals mainly to the fans of the vampire subgenre of fantasy/horror; D&D is built with "hooks" for a wide variety of tastes, both in terms of subgenres (wizards, hack-n'-slash, indiana jones exploration, politics, good-vs-evil, even horror/vampires and steampunk with the right sourcebooks) and in terms of flavour (homourus, serious, heroic, horror and so on).
3) Fantasy is, IMHO, more popular than sci-fi; so D&D gets better sales than, say, Traveller due to this fact.
4) Quality control. The basic books (atleast the 3.0E ones, I don't have 3.5E) have very few typos and a relatively very low level of errata, and were extensively playtested. Compare this to whole editions of Traveller (T4, for example) that died simply because severe errata problems and insufficient playtesting.
5) Marketing. I am no expert in this field but it seems to me that WotC have avoided several of the pitfalls in which many of the competitors have found their death (or, atleast, fall in sales), such as being overzealous with copyright enforcement (it killed TSR), bad quality control (it killed many games), writing yourself into a corner (i.e. focusing only one one niche market) or rule overcomplications. SRD is IMHO a VERY good starategy - it allows and even encourages other companies to publish their own compatible products for this system, which means that a vast wealth of material is available to players (and thus supporting a very wide variety of player tastes). SRD also allows fans to make their own material within a relatively fair legal fraimwork.
6) Playability. While D&D game mechanics aren't the easiest out there to use, they are definitely learnable and playable by most players, even new and/or young ones; rule overcomplexity was a major problem for other systems, as many players got scared or bored by overcomplex rules.