A tale of two RPG heartbreakers...
I. Harnmaster and the Harn Campaign Books (Colombia Games).
This is quite possibly the most beautiful game world ever produced (lore, maps, internal consistency of a real Medieval Europe that happens to have Magic), unfortunately the rule system that supports it is, well, overly complex and encourages a very conservative playstyle. It does not help that it is primarily a skill based d100 % system (yes, you have attributes/ability scores, but they are not super useful). Combat? Hahahahahahahahaha. If the combat doesn't kill you, the infection rules and the limited healing available to the players will (Note: This was the case with 1e Harnmaster, this may have changed with 2e and now 3e were created after the original creator Robin Crosby passed away). It was just plain better to avoid combat if you could, unless you had the jump on your foes with overwhelming force. This game molders in a box in my basement. This campaign world became my go to "lonely fun" reading because I could not find anyone who would play it and at the time AD&D 2e was just to fiddley to make it work with the campaign assumptions. I occasionally pull the local maps (city and site) to use for D&D (whatever edition I happen to be using) but those are game mechanics neutral.
II. GENESYS (Fantasy Flight Games).
FFG (Asmodee) have interesting licensed and house IP that could be exploited for an RPG. Enter Genesys. FFG thought that their Android: Netrunner Living Card Game, X-Wing starship minis combat game, and (at the time) unreleased tactical Fantasy minis game (a Warhammer contender) user bases could be converted into a house RPG system that would allow you to play in a lot of different genres. Genesys not only required you to learn a new system but also by custom dice that were only usable by the game. They were not numbers but abstract symbols that you had to memorize the meaning of to play the game. Who wouldn't want to play in the Cyberpunk future of the Android universe or Star Wars license? Unfortunately, the user bases of the existing games did not cross pollinate like they thought it would and in the case of Star Wars the new system was a barrier for entry for those that played previous system under West End Games (d6s?) and WotC's d20 system. I don't think it lasted a year before development was cancelled.
In both cases, complexity + different mechanics = death. There were not enough players, that pesky network externality, to sustain either as a viable system. Even the lure of Star Wars was not enough.
These both happened before the power of actual play and livestreaming made the onboarding process for new players and GMs all that much easier. Even so, the keepers of D&D recognized the value of this to explode the hobby (even if it surprised them at first) and the rules lighter base game of D&D (in comparison to Pathfinder) and made it the dominate force for TTRPGs on YouTube and Twitch. I don't think Harnmaster or Genesys would have fared well against D&D even in this new environment as they both made Pathfinder seem simple.
With all this said, without D&D none of these games would have been possible even if their survival as a viable TTRPG had the odds stacked against them.
For those wishing WotC and D&D to crash and burn so that something else would pop up in its place, if such a thing were possible, would find the likely successor would be some variant of d20 D&D (Paizo comes to mind) and the whole process would begin anew as that game converted D&D's externalities into it's own. Something as innovative as Fate, Blades In The Dark, or even Call of Cthulhu would have a hard time converting a bulk of those folks over to them. Sure you would see a brief rise as people cast about for something else but most would gravitate to what is known.
One last thought, indy games have a perfectly viable niche in the TTRPG market. They do not need to have the scale of success that WotC or FFG require for a game to continue. However, anything that they innovate making them stand out and has people taking notice, D&D, can and will, quietly incorporate it into their system (sometimes with good effect). For example, the concept of Fronts from the Powered By The Apocalypse game engine is easily transported into D&D. I actually was introduced to that game system by a designer blog that Mike Mearls wrote about incorporating them in your D&D game.
