Well in 4e's case, it was obvious that 3.x's basic assumptions were not lining up with the game people were playing. DM's didn't want to feel forced to fill game days with a given budget of xp or set number of encounters- sometimes your "day" consists of traveling down a road and maybe a single random encounter!
Trying to manage "per diem" resources, then, became a real hassle, and open-ended, powerful spells were so far out of line with what non-magic characters could do that it was a frequent complaint.
Add onto this complicated NPC and encounter design, magic item creation rules that proved to be far more troublesome than they were worth, WBL being difficult to enforce in a non-gamist fashion, people taking multiple Prestige Classes and ala carte multiclassing for front-loaded classes, and the fact that low-level adventures were difficult to write and play, when even a Fighter could be taken out in a single hit, and the necessity of making healing resources the "job" of a small number of classes (instead of doing other things with their resources), made WotC believe that most people weren't playing D&D the way they thought people would- the very thing that would necessitate a completely new line of thinking.
After experimenting with things like the Tome of Magic Warlock and the Book of Nine Swords, they hit upon the idea of balancing the game per encounter more than per day. This way, running a game day with three encounters wasn't noticeably harder to do than one with six. Healing Surges made it possible to not require a dedicated "healer" as much, and so the DM only needed to worry about how many surges would be used by their encounters. Short, 5 minute "breathers" to recharge abilities didn't interfere with the narrative much either.
Stripping away a lot of the narrative power of spellcasters and balancing classes against each other (but keeping out of combat magic in the game, gated by time and money, and available to everyone, not just a few classes) was a way to avoid a lot of the abuse of high level magic.
I could go on, but we all know how this ended up. It turned out that there was a sizable contingent of players who weren't complaining about the problems 4e was solving, either because they were running traditional games, or they enjoyed the gonzo levels of power and customization 3.x afforded, and didn't care so much if playing a Fighter 12 was strictly inferior to a Fighter 2/Sorcerer 6/Dragon Disciple 10/Eldritch Knight 2 (let alone even crazier things like Incantatrixes or Planar Shepherds), along with all the players who hadn't jumped onto 3e and were certainly not going to accept 4e.
If WotC could have developed a "3.75" and 4e simultaneously, that would have been great, but either they lacked the resources to do so, or were afraid of having multiple game lines competing against one another.
But one thing is sure- the issues of the previous editions of D&D spawned from it's core assumptions not lining up with the game people were actually playing can only be solved by changing those core assumptions. 5e still has a lot of those issues.
This is fine for those who are either used to them, or play a game in line with those assumptions, but isn't so great for those who do not. What the end result of all this will be is uncertain, but a game cannot serve multiple masters.
Now some will say that they're fine with games with different assumptions existing, as long as they are not called D&D, but there's no way anyone is going to shuck a storied and valuable franchise IP to make some other game, and if more people want something different, at some point, any company has to go where the market is.