I'll use your questions as a springboard for a tangent, if I may?
One: Was it necessary for 4E to abandon the core gameplay of D&D in order to solve those problems?
No, but it was necessary to abandon core gameplay to allow WotC to sell D&D to new or lapsed customers. The existing core gameplay wasn't bringing in enough customers for WotC to run D&D as the business they wanted. So they had to change the product.
Two: Was the player base's dissatisfaction with those problems sufficiently large and sufficiently universal to make abandoning the core gameplay of D&D palatable for them?
No, but I think that had WotC tried to sell the same game, to the same people, just with some tweaks to solve those problems, it would not have gone well for WotC.
Paizo can do that, i.e. sell basically the same game but "fixed", with the same content again (basic rules, game master rules, monsters and so on), since they are viewed as saving the game. They aren't hoisting the same rules on gamers once again to make a quick buck, so to speak.
EDIT: the very recent announcement of Ultimate Combat is to me an indication of this. Had WotC released yet another iteration of the Ninja and the Samurai for 3.x, the reactions would have been very different from what I expect Paizo is getting from their release.
It is my belief that had WotC tried the same, the backlash would have been significant and the business results would have been unsatisfactory for WotC.
Note that this doesn't mean that I believe that 4e was the only way to do a transition to another set of rules or another type of core gameplay to avoid the "you're selling the same game again, fiends!" trap.
Cheers!
/M