If I read your question as "why has d&d stayed the big dog and not been supplanted by something else"
In part it is the "period" supported by the game. Almost no one has a visceral opinion on how pseudo-medieval (x) works. Whether that's how much a loaf of bread would cost relative to wages (economy), how far a longbow reaches (physics), how mobile a person is in plate mail (stealth), how far a horse could feasibly travel in a day or a human could carry in a backpack (transport), the whole shebang is basically in a time box that is eminently hand-waivable.
Anything modern or near modern (say +/- 100 years) has more issues with plausibility and suspension of disbelief. In some ways its worse because a lot of people accept "movie physics" but others won't, and in a lot of cases it's relative to areas of personal experience. Finding an acceptable AND entertaining middleground for mechanics is hard. Now add the much greater expectations on how the socio-economic part of the world works and it adds another level.
Modern-ish games also wind up tied to sub-genres. Horror, action, superhero, urban fantasy, western, pirates, post apocalyptic, steampunk, indiana-jones type exploration, etc.
D&d is so unmoored from reality you can do all of those. Have a wasteland to the west, an ocean to the east, a demon to the north, a jungle to the south, and a giant city (which is essentially the Greyhawk setting) so you have apocalypse, pirates, horror, exploration and urban fantasy. Add a gnomish city for steampunk, a grassland of nomadic horse archers and invading settlers for old west, etc, etc.
It won't do all of them well but there's less call for verisimilitude when you have them in the same setting and, to be honest, most people don't want to only play one genre so it lets a more diverse group or an easily bored gm change things up without actually changing games and having to rebuild a party dynamic.
The rules since 3e have also ranged from approachably easy (roll a d20 and add one of these numbers from your sheet) to layers of complexity (your summon can flank, there are bonuses from magic weapon a potion of Heroism and the Enlarge spell plus the champion class feature and that special attack feat means you roll two d20, keep the high one, add d4 plus your attack bonus minus 5 and if that is an 18+ on the 20 and your total exceeds their AC you get to roll double damage dice plus 2d4 + weapon damage bonus plus 10 and your weapon does knockback) it can appeal to a wide range of players at the same table.
The lack of tight integration between "D&D the game" and "the setting of this specific campaign" mean a lot of dials can be tweaked to suit individual group preferences. Magic items can be common or nearly nonexistent, classes may not exist, the base society is totally interchangeable, gods can come and go, etc, etc etc.
And it's a space that is hard to compete with. Do something similar and it's a d&d ripoff game, why bother? Do something too different and it risks falling into the subgenre trap.
I honestly think the success of any d&d-alikes are often due to WotC mis-steps where they alienated large groups of players, creating instant market bases. It's hard to dethrone the king when it's popular.
By the same token, d&d has been through effectively 4 major corporate ownerships (Gygax, Williams , WotC, Hasbro) which has caused several pivots and infusions of cash/marketing so it hasn't ossified.
