First, I'm sorry if I lose the thread of the discussion. I'm still at extended vacation and I'm very rarely sober and/or well-slept.
Why is D&D the gorilla in the room, instead of Shadowrun or Call of Cthulhu or Pathfinder or one of any number of other games that are, design-wise, more or less the same? I don't think marketing is the answer; D&D has a history of crap marketing. I think it was there first, colonizing brains, and none of those other games are really different enough to overcome that basic fact.
To put on a tinfoil hat, I'd say the fact that D&D doesn't really support
anything is intentional exploitation of its cultural cachet. An average person probably heard of D&D and didn't hear of any other TTRPG, so the fraction of them who then try playing D&D don't see how heroic exploits of a rag-tag party of misfits is any more supported by it than running a catboy café, and conclude that game design doesn't exist.
More seriously, creative expression (storytelling) and skill expression (overcoming challenges) are at inherent odds with each other -- pursuing one automatically undermines the other. If you engage in Desert Storm levels of planning in D&D and then flawlessly execute an operation where you behead the dragon before it can even blink, like, yeah, cool, but the story you end up with will be... Bad. Like, you wouldn't want to see it on a silverscreen or read it in a book, because "heroes did everything right, didn't suffer and had no need to sacrfice anything" is just damn boring.
And you won't even be able to retell it as a story of your exploits to people who don't know your GM, yet alone to people who aren't familiar with D&D -- it would ring hollow. Yeah, cool, dude, you killed a dragon, so, anyway, I had this dream last night where Tilda Swinton was choking me...
There's this joke about the police here: "The fact that you are not in jail isn't your merit, but our flaw", and well... The only reason the 1st Tarrasque Cavalry Regiment of 3rd Archlich Army isn't hiding behind every corner is because the GM decided she doesn't want to put em there. She could. She didn't, but she could.
In a sense, GM is basically designing a game that only you can ever play, and other people will have no frame of reference to understand the significance of something. Yeah, the only reason why every enemy in Dark Souls isn't Ornstein&Smough is because From Software decided they don't want to put them there, but could, but other people can just go and experience the same game.
But if there were hard, strict rules on what GM can or cannot do in D&D, it would only enhance both the gameplay and...
Another way of looking at this is that D&D's potential flaws are actually behind its appeal. It has enough story to give a sense of purpose and continuity to the session, so that players want to keep coming back. They want to continue the story. It has enough gameplay elements so that players can engage in creative problem solving within various tactical and narrative restraints, and can aspire towards improvement if not perfection. Could it not be that D&D's "jack of all trades, master of none" design manages to strike a happy medium? It is, at its heart, a kind of half-assed game, and my conjecture is that Arneson, Gygax and co. kind of caught lightning in a bottle with this sort of game structure, in ways that work really well to trigger more or less addictive reactions in human brains.
...the story. Well, in a sense.
As I said, storytelling is fundamentally incompatible with overcomming challenges -- you either provide outcomes based on what makes the story good, or you don't. You can't have more than one first priority.
But if you embrace that the story is more or less linear and the "meat" of the game is, well, the game, it can enable greater intentionality in storytelling too.
I really enjoy Dishonored 2, now that I know how all endings are achieved and can deliberately work towards the one I want.
For a TTRPG example, right now I'm playing in a Dark Heresy campaign that uses an adventure I've already ran.
I know what is actually important to the plot and what isn't, what choices I'll have to make anyway and what choices lead to a branching path, and that allows me to express my character better — because I know the boundaries in which I can do so.
There's a segment that is pretty much a big long cutscene and I know that it's a big long cutscene where nothing of note can be done — so I don't even try to do anything of note and can focus on little things. Is my gal humbled and awestruck by a glorious cathedral devoted to the God-Emperor and will spend her time there in a constant prayer, taking the chance to attone for all the previous and future sins in a holy place where He might actually listen? Or is she wandering around in her tasteless but expensive garb, admiring lavishly decorated building with a sense of amusement, giggling at the reactions monks give her? Or maybe in her paranoia, she will suspect everyone, even in this sacred palace, of heresy and fake faith, looking for clues that aren't there?
Ultimately, it doesn't matter. PCs will be just handed the next clue in the morning, but expression is still cool!
Another player, who didn't know that it's nothing but a cutscene, spent the whole session trying to look under every rock, and was very frustrated he couldn't find anything. If the GM just bluntly said that he'll never find any traces, regardless of his actions or rolls, I reckon the guy would have a much better time.