1. It's about what the rules don't support. The negative space in D&D (the places the rules don't exist) is actually an important feature, not a bug, of the game.
I believe I can see what you are championing here, let me restate this to see if I understand it.
"Because D&D gives authorial control to one person, who therefore requires the trust of the players and a certain set of skills (heh), areas unsupported by mechanics are still open to skillful adjudication and play as befitting the table."
I think this is where you are going, if I understand I'll dive into it, but I don't want to accidentally go off on a tangent from misunderstanding.
2. It's about the community of D&D. Right now, I can play a game of D&D with a grognard who is running a highly hacked version of OD&D with a 200-page player supplement that, inter alia, details how he allows the original White Dwarf Barbarian as a playable class, or I could play D&D with a group of high-schoolers that use a lot of 5e homebrew to make it an anime/manga game with Wuxia influences. And all of those people are communicating and cross-pollinating in the greater D&D community- something that no other game has, and no other game is even close to having (in terms of both size and history).
This is nothing inherent to the rules, and is inherent to the brand. As a counterexample, let's look at 13th Age. It's a d20 OGL (so D&D bones) written by one of the lead designers of D&D 3e and the lead designer of 4e. That's got to be closer to many than older editions, kitbashes, and heavily homeruled games
as a system. But it doesn't have the support of
the brand. Without the name "Dungeons and Dragons" emblazoned on it, it's not a big part of the "greater D&D community". So really, this is something 5e has going for it, but not because of anything between the covers of the books.
Now, this goes back to you saying "system" and I using the term "mechanics". Perhaps "system" does include the brand and these other aspects, but to me they aren't part of the rules, but almost are in spite of them.
And then you have edition wars and D&D players who fight even more than they do about other systems.
3. It's about the flexibility within the game- this is, perhaps, the most important. Building on what
@Gradine posted (re: types of fun), D&D has a long history of being able to engage different kinds of players, seeking different kinds of fun,
at the same time. On these boards, we often hear about groups that try another game for a while, and then "return to D&D." I suspect that this is because a lot of games are built to
primarily appeal to certain kinds of fun, but are not as engaging for all types of players. Take
BiTD, for example- it's a great and brilliant game. But players that are really into narrative, discovery, or abnegation (yeah, I don't like submission either) ... maybe not so much?
It's funny, I add up the same points and get a potential negative. It's about what a table wants.
If I want a fast sports car, and my friend wants a all wheel drive pickup, and a third friend wants a minivan because they have four kids and it's just the easiest, looking for a compromise of a sporty SUV may be something they each could settle on, but it's not the best for any of them.
Even within people looking for D&D, I know some who name their characters "Burger King" and have silly adventures fully of punny names and Monty Python references, some who are looking for heavily tactical chessboard battles, and some who are looking for character drama soap opera where adventuring is the genre and backdrop. They likely wouldn't have a good time all sitting at the same table even though D&D can -and does - cater to all of them. If your table is out of sync with each other in terms of expectations, just haing a system that can cover won't make everyone happy.
And once you are looking at a unified table, picking the Porsche instead of the sporty SUV can provide an even better experience.
Now, to be honest, a unified table doesn't mean unified in all ways, and in that variation the flexibility of a big tent game like D&D can be a good fit, assuming that the players are looking for a D&D-like experience in the first place.
And getting back to brand, it's likely the most common first RPG, and for many only RPG that they have enough players to actually play extended amounts, so there's a huge community that thinks that the D&D feel is synonymous with all RPGs.
As an aside, I grew up with fake maple syrup. Here in the States it's high fructose corn syrup (read: sugar) with corn syrup (read: more sugar) and artificial maple and butter flavors. It tastes very little like actual maple syrup, which I didn't have until my late teens. But I buy it today because it completes the comforting taste of my childhood pancakes and waffles. But I don't mistake it for
good.
And that's where D&D (esp. 5e) can shine. Yes, the drawback of it, the failure point, can be a bad DM.
And that's one of the historical holdovers that can be the downfall of D&D. You talked about how the video game industry has advanced much faster than the RPG one, but we have come up with points that make running a game a different all-powerful-so-must-be-good experience of D&D. There are games with very clearly defined GM rules. And DMs used to D&D where they can change rules as they wish don't seem to get that this is a massive shift. Here's how the DM does their part, just as defined as the player side of the player loop. Here's what they must do - they don't have lassitude. Here's what they can't do - they also don't have lassitude. Here's the ranking of what you need to consider when making a call.
I've played with new RPG players and had a great time - as great as with experienced players. But with D&D you can not say it is the standard that a newer DM will provide for the table as good an an experience as a veteran DM. But other systems have reduced that bar and redefined the role so the table as a whole isn't nearly at the mercy of the quality of GM.
When you can say that you (the plural you) have a consistently enjoyable play experience with a mediocre DM, then D&D has caught up to industry. Until then it offloads so much of the game onto the DMs shoulders that it can be a critical failing point. And a daunting barrier of entry for new DMs.
D&D is not a great game system for everything, or every person. But it's a really good system for a lot of people when you have groups with mixed interests. IMO, etc.
D&D is a quite good system. Everyone should play it at least once, at the very least for context for that "greater D&D community" you spoke of. But there are also experiences that other systems handle very well that tables interested in them should also try - likely more than one. There isn't a system wide enough to be everything to everyone. So I advocate multiple systems - find what fits best for the feel your table wants next. And then feel free to change for the next.
