No one framework does everything well. The d20 framework, for example, would not have done the Deadlands campaign I just ended well at all.
The d20 framework is mostly just slightly cleaned up D&D scrubbed of few proper nouns. Of course it doesn't do a lot of things well.
There have been some good universal and multi-genre systems produced, though, that can handle virtually anything you might want to base an RPG upon. They'd constitute that kind of 'framework.'
So, yes, there is a desire for different frameworks, because not all players like the same things, nor does even one player want the same thing all the time.
Very true. Even though we've had some ideal 'framework' systems put forth, none has achieved dominance, nor even much success. That's simply not what the RPG community wants - they want D&D, for D&D's sake. There's a hard-core of Forge types who want to find the 'perfect' RPG, and the kind of 'framework' idea mattcolville came up with would be a candidate for that. But the bulk of the hobby started with D&D, and either still plays D&D, rarely plays at all, or comes back to D&D when they, inevitably, can't find that elusive 'perfect game' (or, tragically, find it, and can't find anyone else who wants to play it).
In diversity, there is much more innovation and flexibility.
Nod. You can see that in the indie side of the hobby vs the D&D/d20/Pathfinder/OSR side. On the indie side, it's all about innovation, and a revolutionary game can see print and possibly not bankrupt the guy publishing it, and possibly even be remembered years later. On the D&D side, familiar D&D-ish-ness sells well (and dominates the hobby, when combined with the D&D name), while the corpses of fantasy heartbreakers who were D&D-ish but for an ambitious change or improvement or two, litter the streets, and even D&D doesn't dare improve or innovate by more than the barest increment at a time, for fear of a backlash like the edition war.
It doesn't have to. If it does enough well, and satisfies enough people, then there won't be enough demand for other frameworks, and you'll never see a competing framework at the same level of success.
I think this is what D&D has done. There's demand for other frameworks, but not enough.
Not remotely, no. D&D is a terrible framework for anything but it's own demented melange of fantasy/sci-fi/lovecraft - and not even very good for that. d20 is not much better. It's just not a mechanically great system - which is understandable, since it was the very first RPG, and changed very little for the first 25 years it was around.
But, in addition to being the first RPG, D&D is the only RPG anyone outside the hobby has ever heard of, and thus the most likely first RPG experience for new gamers testing the waters of the hobby. So, anyone who tries D&D and doesn't like it most probably concludes that they don't like RPGs, and never enters the broader hobby. While anyone who stays with the hobby, because they liked their first D&D experience, is likely to stick with it or at least come back to it now and then.
The problem is that while almost every supplement has something worthwhile in it, it also has a lot of crap.
Sure, thus 'system mastery.'
If you follow the game from the start, you can have a pretty good idea of what's what because you went through the books with a few months in between each one. But if you start playing when there already are a bunch of books out, that can be pretty intimidating.
Also very true. Which is bad for growing the hobby by retaining new players, but good for serious players who develop lots of said system mastery. Pathfinder is succeeding on that phenomenon. 5e's slower pace of releases might indicate a hope to retain more new players by keeping the game more approachable, longer.
You might be able to mix Conan, Elric and Lankhmar, but they're all in the "Sword and Sorcery" sub-genre of fantasy. Mixing in Tolkein's high fantasy, Vance's Dying Earth science fantasy, Arthurian romance and 19th century Horror gives you something that isn't part of any particular genre and won't be identifiable as one. Which is what D&D did, taking a set of skirmish wargame rules, sticking in influences from a huge variety of fantasy that seemed cool at the time to the participants, and created something that wasn't really part of any of those sub-genres but either one of its own or just Fantasy in general. Which is what hasn't happened in SF RPGs, with no game doing the same thing in taking a wide spread of influences and incorporating them whilly-nilly into a rules set that wasn't particularly designed to handle all of them.
That is what D&D did, but I think it's probably more likely a strike against it than a reason for its success.
Just as the relative failure of great 'framework' presenting games like GURPS and Hero argues against the idea that D&D is succeeding by providing a 'framework' that people are using to play many different games, there actually /were/ science fiction games that went the whole-hog approach, early on. They weren't all Traveler, d6 Star Wars, or Battletech, with a liscence or single genre or single original setting.
Space Opera is the one that springs to mind. I think it's pretty obscure and was never that successful, but it took the whole universe of sci-fi literature (from HG Wells to EE 'Doc' Smith to Gordon R Dickson) and film/TV (Flash Gordon to Star Trek & Star Wars) as it existed towards the end of the 70s, and stuffed it into one perplexing, D&D-level-of-genre-incoherence, RPG.
Apart from the iron dominance of D&D, sci-fi RPGs have never been all that successful because of a critical difference between the sci-fi and fantasy genres. In fantasy, you ask the reader to accept a somewhat standard, familiar set of preposterous assumptions: magic, heros, monsters in an ancient or medieval setting. Fantasy works and sub-genres vary, but they don't vary wildly in that formula. The art of the science-fiction story, OTOH, is to ask the reader to accept /one/ preposterous assumption, one 'What if?' and the story flows from that. Earth is invaded by Mars. Positronic brains make sentient robots a reality. FTL travel is invented. Etc. That one assumption or question makes each sci-fi setting/story different, and that makes munging them all together the way D&D did with fantasy untenable.