Its accidentally on purpose.
D&D fans want casters to have certain characteristics.
But the fandom didn't realize that is is only balanced under a rather rigid playstyle. So it is easy to change or miss one thing and "Oops Casters are broken".
Yeah, I find this to be the reality: it wasn't intentional to underbalance spellcasters and martials, but there was a conscious decision to make spellcasters good and to go back to the things that made spellcasters so good in previous editions. They may have thought they balanced it in other ways (for example, the addition of Concentration), but it doesn't end up doing enough to offset the massive game-changing abilities out there.
Spellcasting should not be instantaneous. It'd be a lot more strategic to use if you had to wait for the start of your next turn for your (levelled) spell effect to go off.
So the whole Greyhawk Initiative was a cool idea that made this easy to incorporate. When your actions affect your speed, suddenly spellcasting becomes a choice. I made modifications so that Cantrips were a d8 and spells were a d10... but they only
started at that point. They would be finished casting a number of initiative slots equal to the spell slot used to cast the spell (Alternatively if you want upcast spells to have a bit more use, you could have it cast after a number of initiative spots equal to the spells natural level, meaning Magic Missile at high levels becomes an interesting trick). This meant that people could potentially attempt to disrupt the spell or even get out of sight of the spell, forcing a spellcaster to hold onto a spell until they have a target (similar to how someone can delay a spell under the current rules). I never ran a proper game with them, but the idea seemed sound enough.
Exactly, it's not balance. It's "What level of imbalance do I prefer?" 5e's level of imbalance is too large. For me it's not really large enough. I find 5e balance too confining, but not confining enough to seriously impact my enjoyment of it, so I continue to play.
It
is balance, though. People are talking about the well-understood idea of balance being something that isn't perfectly achieved but rather outcomes being matched in rough amounts. Trying to pitch that as "imbalance" misses the well-known, colloquial usage of the term. And honestly if you have to resort to these sorts of incredibly pedantic arguments of semantics, you've basically conceded the actual point and are just trying hide it.
As for the actual topic, yes, the imbalance wasn't intended but was inevitable given what they were trying to achieve: a huge overcorrection in regards to turning spellcasters into what they were before 4E while also trying to make martials (particularly fighters) "simple".
Martials just have a lot of in-built problems that the system is not designed well enough to accommodate. For starters, they are much more equipment dependent than spellcasters: more monsters are resistant to non-magical damage than magical, and the only martial without actual magic that can get around this sort of problem is the monk, as their strikes naturally become magical over time. If you haven't been able to get a silvered weapon or a magical weapon, your fighter is completely nullified in a way spellcasters simply won't be: even if you try to restrict finding spells, they can at least gain spells when they level up. They will be choosing cooler and more powerful attacks while martials are very much at the mercy of the GM as to what sort of equipment falls into their hands, or what they can purchase.
And that's sort of the continuing problem with martials: they depend on the good will of the GM in a way spellcasters don't. The spellcaster can break the rules of the game explicitly while martials have to barter for the ability to do so. Could you do cool things as martials? Sure, but it's less something spelled out in the rules and more by agreement of the GM, and at that point that's not about design as much as GMing and adaptation to flaws in the system. And even then, spellcasters can also barter for cool effects as well.
And while balance shouldn't be the biggest driver of design, it should be at least an
important part of design: if you have no balance between your classes, the niches you try to create for the classes will fall apart because some classes will be able to expand into the areas other classes are meant to inhabit, while others will have a hard time actually being able to fill their own specialty. Someone brought up that the designers had wanted certain classes to be less powerful in combat to make up for being better in one of the other pillars, but I think that's one of the biggest failings they have: certain classes should probably be weaker outside of combat given what they can do are not (wizards, bards, most spellcasters really) while those dedicated to combat do not, in fact, dominate this pillar given how much they give up elsewhere (fighters). And then you get the rare class that is meant to get something outside of the combat pillar but everything they get comes off as kind of lame, making them lame in the place they should exceed while still having problem with combat (ranger).
Are these imbalances fixable? Well, yeah, I've already seen games that have done a decent job of balancing these sorts of divides. Can D&D do it? Absolutely...
if they intend to. You can make interesting fighters with more fantastical powers and pull back a bit on the power of spellcasters. I think the problem is that there is resistance to even recognizing that there is a problem, to the point that people are playing word games trying to get around even having the discussion.
