Um, because they realized that you fundamentally can't have a per day mechanic class balanced by a at-will mechanic class.
The assumption is that "a few tweaks is all D&D needs" but for many people, that doesn't fix what they see as problems with the game.
That's of course assuming "balance" is a concern. This seems to be part of WotC characterization of its target audience, and might be why I still fall into it. I don't want to play second fiddle or be limited to a particular class/race to avoid the second fiddle.
But it's not the only concern, obviously.
Variety, different archetypes, character creation/build options is another. 3E had this in spades. 4E puts more constraints on this, but still has this as a goal.
Another concern is "tactical interestingness" - resource management during combat encounters. 3E had this at different degrees- very strong for spellcasters, very weak for non-spellcasters. 4E things that everyone wants a similar amount of options and management during encounters, they just like the different focus (the roles are a shorthand for the directions they identified as important.)
Another concern was "playability" or "usability". How is the game played? How can it be played? How can we remove the parts that people rarely use or find annoying? How can we add stuff that people already do or would like to do, but don't know yet how to? 3E did something in this regard, by trying to streamline certain aspects of the game, but it still managed to get very... complicated in certain areas (monster and NPC creation?). And this also lead to stuff like no Craft/Perform/Profession - not used by enough people, too vaguely defined, seen as a "skill tax" for some classes (Bard). Use at the table is minimal, so off it goes.
Yet another concern might be "flavor/fluff". I am a fan of the 4E cosmology and races, but there are others apparently are not. Still, I think the "implied setting" is interesting and no worse then the 3E ones. but it's not the 3E or earlier editions one.
Maybe here WotC made a mistake, believing that fans were ready for a new implied setting. I certainly wouldn't have expected people to see the lack of the Great Wheel as a problem, because I never experienced it as something "core" to D&D (especially after Manual of the Planes showed alternative cosmologies).
Of course, I am not sure how much all of this was internally written down as design goals based on marketing research, and how much can be attributed to the preferences of the design and development team (and their ability to find compromises between their preferences.)
The priorities that have been assigned to these concerns have changed between 3E and 4E. And thus 4E became a better fit for gamers with shared priorities. And now many 3E players find out the new priorities are more to their liking, and many others find out they had different priorities.