Leaving aside 4E for the moment (I agree that it was the most ambitious attempt to break away from TSR legacy design), 3E had the unified mechanic of "D20 plus modifiers against a DC, roll high", but "streamline" otherwise seems a bit off to me.
Character and monster design was at its all-time highest* for complexity and detail. Skills and feats and the combat system had massive amounts of new rules*. Spells and magic were much cleaned up from AD&D, but at least as detailed.
I do think it mostly built onto AD&D rather than trying to break away from it. Comparisons of spell verbiage, for example, between 1E and 3E show a remarkable amount of overlap. And perhaps that's more what you're getting at. 3E was faithful to AD&D in many ways despite switching to the D20 core mechanic and adding a skill system, feats and so forth.
(* I recognize that some of this was prefigured in the Skills & Powers and Combat & Tactics expansions in late 2E, but to the best of my knowledge most of that was never widely adopted, and none of it was part of the core rules).
Yes, true - but it was, in my view, an attempt to update D&D to a more contemporary context. And as I said earlier in this thread or another, a lot of this was Jonathan Tweet, who worked on Ars Magica with its clear dice roll + attribute + skill mechanic, and also Talislanta, which also had a really clean and simple d20+ mechanic.
By the end of the 90s, D&D was a rambling, anachronistic heap - loveable, of course, but messy. I think the main impetus behind 3E--from a design perspective--was to clean it up and update it. And of course with that core d20 mechanic, we had the d20/OGL "revolution."
wrong
4e outsold 3e
4e only didn't hit a made up number that WotC wanted it to (not only higher then 3e 2e or 1e but substantially) and they saw that for the first time a competitor didn't have a new system (like WW in 90's) but was basically holding out an older edition as competition (something no edition before 4 delt with) they saw they could raise there profits (hey it is what it is and we all need to eat) by selling the forward progress of 4e out for nostalgia and it worked (although the new edition of PF pushing more away from 3.5 helped 5e too)
100% agree 4e was a start not an end. If 5e had improved on 4e I think we would be in a better place going into 2024 (IMO) however we may not have pulled back 3.5 fans, and as such even with growth would not be as big as it is.
I honestly think that it all comes down to Piazo. if PF never was 4e would have exploded. Now I can blame wotc twice for this (They made the OGL... then they didn't include it)
It isn't only about raw sales, though - it is also about the cohesion of the D&D community. 4E fractured the base unlike any edition before or since. And I wouldn't blame Paizo - Pathfinder was a response to 4E's mixed reception, not a cause of it. In other words, they filled a need: for folks who were basically happy with 3.5 and wanted 4E to be a revision of it, not a revolution.
And of course this is supported by the fact that 5E essentially brought the band back together, and a large part of that was through going back to a more "traditional" version of D&D, at least relative to 4E. Pathfinder diehards kept their subscriptions, but I'm guessing a lot of them were more amenable to playing 5E than they were 4E (not to mention retro-cloners, etc).
When 4E came out I ran or helped run a couple of game days in a major metro area. There was a fair amount of enthusiasm even if we did lose a lot of people to Pathfinder initially. But slowly over time people just faded away. We went from an initial peak of probably 8-12 tables twice a month (so around 50-80 players) to a couple dozen die-hards. So it wasn't "cultural forces" or "minor celebrities" or even major celebrities. In our area 4th edition started relatively strong despite losing a significant percentage because it did initially bring in new people. At lower levels it was a more approachable game. But ultimately it simply didn't have staying power for mass popularity, unlike what we have now.
There's no telling what could have been of course and lack of broad popularity and staying power is not the only way to judge game design. I just don't think 4E could have ever approached the numbers we see with 5E, even if some of it's design and intent was an improvement. If it had been it's own thing, I suspect it would still have a decent following. But it's all just speculation and guesswork at this point.
With regards to cultural forces and celebrities, I was referring to 5E. There's still a mystery of how 5E became the phenomenon that it is, and I'm saying those are factors. Back in 2008, there was no Critical Role, no sexy True Blood stars sharing videos of D&D uber-rooms or having Tolkien geek-offs with Stephen Colbert. And, I would argue, there wasn't the same thirst for it in the younger folk - they were happy with their Nintendos and iPhones and Warcraft. Fast-forward to 2014-18 and we have a different cultural climate - for whatever reasons. I'm not saying that the celebrity factor was primary, but it helped - and was part of a rising tide.
Clearly we'll never know, but my point is that if A) the bulk of D&D's current base are people who are new to it with 5E, and thus younger, and B) there was a cultural/generational thirst for an organic game of story and imagination in 2018ish that wasn't there a decade before, and C) most of the folks who had a problem with 4E were older/long-time players, then D) 4E might have done just as well as 5E, at least as far as the new generation of players is concerned.