So I game with mostly people younger than me, by 10 years or more. What I've found is that there is a mindset, deeply ingrained into younger people, that older things are not as good as newer things. I blame the relentless gears of capitalism for this, as companies are constantly trying to sell you "newer & better"- even when "newer & better" doesn't live up to their hype (I still think Windows NT was superior to the dreck that came pre-installed on my new PC).
Just look at Hollywood's incessant stream of needless reboots as an example of this in action. Some years ago (in the before times, pre "global situation", as I've been asked to call it online), the older hands in my group had a problem with a newer player. We'd mention great fantasy movies that he should really watch, and he'd always refuse, acting dubious that anything old could be good. Somehow, the movie that became the biggest point of contention was The Princess Bride. Anytime we'd quote the movie, he'd get a confused look on his face, annoyed that he was locked out of the joke, but refusing to give the movie the time of day.
Finally, after much cajoling he said "if the movie is so good, they should put it back in theatres. I'd watch it then". Now, as many of you no doubt know, this blessed event actually happened, and he succumbed to peer pressure.
I'm happy to report he had a blast, and admitted the movie was quite good.
But the fact that it took so long to get him to see it is troubling. And D&D is no different. Younger people will hear tales of "Thac0", "Hulking Hurlers", "4e is just an MMO", "Housecats can kill 1st-level Wizards", and focus on that, no matter how much you explain the positives of previous editions. It comes down to "people say it wasn't good, so it must be bad, or there wouldn't have been a new edition"- it's like the flipside of the "it's popular so it must be good" argument!
Now maybe, if TSR (and later WoTC) kept reprinting the same books over the decades, with only minor tweaks (like how most book editions work), people would still be playing the older game. But I know that, by the mid-90's, D&D had gotten a reputation for being an archaic dinosaur of a game, as people were quickly moving onto the new shiny games like Vampire, Shadowrun, Earthdawn, etc.. (whether those new games were better or not is a matter of opinion).
So in the name of Keeping The Lights On, it was decided to massively overhaul the game for the new millenium. And pretty much set the precedent for what a new edition is (not necessarily what it should be).
Now we reach today. 5e has been out for 10 years. The game has changed a lot in that time- 5e as it was in the first 5 years isn't the same as it is in the last 5. 5e didn't have to be anything more than "not-4e". No real expectations were placed upon it beyond "get old and new players to buy it"- if this ostracized people who actually liked 4e, well, too bad, we gotta get back the 3e fans, the TSR fans, and the Pathfinder fans!
But to sell everyone new books now? That's a bit harder. You want to get the people who didn't jump aboard 5e, keep the people who did jump aboard with early 5e, and keep people who jumped aboard with later 5e- all of whom have very different wants & needs.
Also, probably a good idea to change things enough that it isn't a repeat of 3.5, where it gets the reputation of a cash grab. So how to proceed with this impossible task?
Well, you'd probably be best off realizing it's impossible and not trying, but alas, your parent company's shareholders demand more money, which means new books.
Any changes you make/don't make are going to annoy people (this thread alone has evidence of that!). I don't know what the actual discussion was at the company, but someone had the bright idea of shifting the blame onto the players with another open playtest! But unlike the last one, we can't just shut the game down for 2 years, no no, we have to keep our current customers on board!
This may be why we got the "don't worry, it's the same game/it's going to be backwards compatible" claims. I don't know, I wasn't there. And when the final product isn't what everyone hoped it would be, well, "hey we did market research, and it says this is what most of you wanted! It's not on us!"- again, wasn't there, don't know for sure, but it sure sounds believable to me!
Because it really was an impossible task. There's no way to keep everyone who loves D&D happy without making someone else unhappy. And some of those people they are trying to please simply cannot be swayed without setting the clock back to 1984! Meanwhile, you got to make the game look "new & improved" for the next generation (again, whether it is or not).
We've come to the point where there is no good answer. Make D&D a modular omnisystem? New players and DM's will be confused about what version of the game would suit them best. Make multiple D&D games? More of the same, and you might end up competing with yourself! (I mean, how wants to play "Basic D&D" when there's "Advanced D&D" out there! I don't want to be accused of playing "kid D&D"!*)
*I'm wildly aware that wasn't the case- non-AD&D has quite a bit of complexity under the hood, but between the D&D cartoon and Morley the Game Wizard, the marketing would certainly have you believe such lies! Younger, stupider me fell for it, to my shame.
If you feel modern D&D isn't for you, there's no life preserver for you, I'm sorry to say. You either have to make do with the older D&D, alternative D&D from 3rd party publishers, or some kit-bashed "looks like modern D&D but totally isn't".
Or you wait and hope the pendulum swings back in your direction 5-10 years down the road.
And in most of these cases, you may have to deal with newer players turning up your "old person D&D"- after all, if it was so good, then why is there a new edition?