My feeling is there are 2 classes of things that 4e did WRT 3.x.
Some things fall into the category of "we just wanted to create a somewhat different game" which could potentially be for a lot of reasons, such as a desire to shape D&D in a unique way, a perception that such a change was a 'fix' to something, simple mistake in design or execution, unintended side-effect of other changes, desire to conform to newer trends in market, etc etc etc.
Other things fall into the "something specific is actually not meeting our product needs with 3.5". There are actually a BUNCH of these, IMHO. 3.x classes didn't work very well. It proved exceedingly difficult to create workable usable classes. 3.x's magic system produced a very specific sort of play, and altering it generally required MAJOR surgery. Some types of play were heavily disfavored. The skill system was borked, and this lead to great difficulty trying to make viable non-combat subsystems. MCing and PrCs badly undermined the class system concept. Trap options were actually MOST of the possible options in the game. Etc.
Honestly, I think 4e actually arose out of a desire to make a core system that would better accommodate 3rd party material of high quality. MANY of its features seem to be designed deliberately to lead developers to make good choices. As a result a VERY high percentage of 4e material is of high quality. Even the 'bad stuff' like HoS is still measurably better than almost any 3.5 material that was ever published in mechanical terms.
None of this is intended to be a knock against 3.x in terms of being the game people want to play, or saying it is 'bad' in some fanciful 'objective' way, etc. It DID have certain very specific quirks, strong ones. I think developers were often frustrated by it.
One of the terrible ironies of 4e is that it was ideally suited to be OGLed and form the basis of a whole series of variations and additions, yet WotC chose to eschew that opportunity, further stunting the game's acceptance and popularity with a very key group of thought leaders, 3rd party developers (who also tend to be pretty active in terms of playing and writing about RPGs).