I just don't think it's particularly good game design...
I understand and know that other people disagree, but just like you (and others with similar views) have no problems engaging me on the issue, I'll engage as well.
And, again, why does one character (the spellcaster) have to be /better/ than another of equal level, especially by one of the most important metrics of the game (Combat)? If you want casters to be better than noncasters, give 'em bonus levels and/or cap noncasters at a lower level. All you get if they're unbalanced at equal levels is that the game is harder to work with from the DM's side and unsatisfying mechanically for a nontrivial portion of the player base.
I guess I don't understand where this expectation originated. I mean, I can't think of any genre fiction where there's any semblance of balance between different different types of characters. Nor am I aware of any other rpgs that emphasize that premise to the extent we're talking about here. The idea that the choice to learn how to use magic or to pursue adventuring without it should be equal in an rpg (not the same, just equal) seems to be almost exclusively confined to a portion of D&D players.
If we were playing Lord of the Rings (a touchstone for most D&D players), no one would complain that wizards and rangers aren't equal, let alone the commoners at the center of the story. But even beyond that, if we were playing Star Trek, the engineer class would be able to do anything with time, the command officer class would tell everyone else what to do, and the security officer class would just be good with phasers. If we were playing Law and Order, no one would complain that the lawyers are better than the police (or vice versa).
That's the story angle, but from a game angle the same is true. In any game that posits meaningfully different roles, those roles are almost never balanced. In any given team sport, some positions are more important than others, and some require more skill or ability than others. In Command and Conquer, some units are better than others. In miniatures wargames, some pieces are better than others.
So again, the expectation that two completely different D&D classes should be equal across all levels doesn't have much precedent. Again, I'd agree that it's a nontrivial portion that has this concern, but I just don't understand where this perspective is coming from, or why it seems to override other concerns so readily. I mean, I like when things are balanced, but only where a number of other goals are satisfied first.
You're conflating balance with sameness and equality. It isn't. Balance is about providing options that are worth taking, not taking away options or making sure everything is exactly equal.
It isn't an absolute, but the 4e concept of class design is a big step towards "sameness". It's not at the point where everyone has the same mechanics and you just describe them differently, but it's closer to that by far than any other version of D&D, and that's jarring. The merits of that step are what we're discussing.
And for a /very/ large portion of the people who have played D&D, combat is a major component of the game - I've never heard anyone talk about a D&D campaign that didn't feature combat as a central feature. If the combat doesn't work, the game doesn't work.
Well, no one's arguing that point. I do, however, wonder if you're implying that prior to 4e combat didn't work (for everyone, for some people, or even just for you). I mean D&D has been going for quite a while without striving for or achieving 4e-style balance, and yet has been balanced in other ways and for other purposes. Was it wrong to do so?
Or are you merely suggesting that incremental balance improvements are part of the natural evolution of the game (which I agree with).