D&D has always been the best-selling RPG. It will almost certainly stay that way. This has little to do with its game design. At best what we can say is that it has always been 'good enough' to not jeapordise its market position.
1. The rules of D&D has changed significantly between editions. The 'TSR era' is very different to the 'WotC era'. 4e is seen as very different from 3e. 5e is sort of a mix of 3e and 4e, the best bits or the worst bits depending on what you value. The game has variously been sold as two simple little books in a red box and three giant hardcovers full of maths. None of this has ever really mattered. Even in the 4e era where it had a very close competitor, that close competitor was selling a version of the previous D&D with the serial numbers filed off, and had a pre-existing direct relationship with the D&D fanbase.
2. D&D is the 'gateway drug'. It's what the vast majority of new players start with. It's what people who don't play RPGs think of when they hear about RPGs. It's what's shown in the movies and the TV shows. It has massive brand recognition and network effects. You can't tell me this is because of a critical analysis of the game rules currently in print, and that if 6e was a rubbish design, or too focused on one style of play, these externalities would suddenly dry up and go away.
I think the only way anyone could ever even hope to dethrone D&D is if you had i) a game based on an extremely hot licensed property that is a craze with the general public (I'm thinking Star Wars and Harry Potter at the height of their popularity), ii) a large publisher that had extremely deep pockets for mainstream distribution, marketing, and outreach to the non-playing public and general media, iii) a very permissive licence (or just embedded within the property owners) so that all the normal difficulties of licensed properties don't interfere, and iv) D&D happened to be at a low point anyway and wasn't able to rescue itself.