What is the obsession with class balance?
Balance is a critical aspect of any game, RPG or otherwise. Class balance is critical to D&D because Class is such a big part of a character.
Why should a magic-user and fighter or rouge and cleric all be comparable in power at the same levels?
So the game doesn't automatically suck for you based on the archetype you want to play.
Isn't balance a subjective quality that can shift based on play conditions?
No, it's prettymuch objective, but, yes, it can shift based on play conditions, if it's not very good balance. A well-balanced game will support a variety of play styles and conditions without breaking.
I think one of the things that did the most damage to class balance was putting all classes on the same xp progression table (I think this occurred in 3E).
Ultimately, experience progression is entirely abstract. If you have one character that gets to level X and Y,000 exp, and another that gets to level X+2 at Y,000 exp, but the two are balanced at those different levels, then that is no different than having two charters balancing at the same level & exp totals. The only difference is that it's easier to tell that characters are of about the same power level if /level means the same thing to all classes/.
I think 4E magic using classes lost that mystique by trying to force balance through the rule set rather than putting that in the hands of the DM.
They did lose a certain, subjective, mystique, yes. That was a very small price to pay, though, especially as the loss is meaningless to anyone who doesn't have the same subjective opinion about 'mystique.'
4e casters still do tons of physically impossible things, it's just that in the abstract math of game mechanics, those things are more nearly balanced, now.
Putting balance in the hands of the DM makes the role much harder. That means more DMs screw up, running imbalanced campaigns that frustrate players and put them off D&D, and many more players simply never DM, because it is such a difficult and thankless job. In 4e, it's much easier to run. I was never able to find more than one or two campaigns of AD&D or 3e to play in at a time. I'm currently in 3 4e campaigns, plus one Gamma World, and I've turned down invitations to others. And, while I rarely ever ran 3e (having run AD&D for 10 years straight, so a little burned out at the outset, and just not having the time 3e demanded due to RL), I frequently run 4e.
It's just a much better game from the DM perspective. If you like the challenge of taking a really bad game, and thrashing and house-ruling it into playable shape, though, it's definitely not for you.
And if you think about it, there was additional balance built into AD&D that a lot of people just chose to ignore (it was too hard to use, slowed game play, etc.).
There was a /lot/ of that, yes. Vancian casting, too was a hard limit - but people made up spell-point systems. Each ed, actually, went and did what people were doing with their house rules: removed restrictions on casting without powering it down correspondingly. Until 4e, which leaves casting no more restricted than non-casting, but also not much more powerful.
I fully understand that we all want to be special flowers at the table with our characters, but should that not be more about character development and roleplaying than stat blocks?
It /should/, yes. Which is why balanced mechanics are better than crap mechanics - because they allow you to make your character about development and roleplay, instead of chasing broken combos.