I think it's important to remember that a lot of the issues people had with 4E weren't just about the mechanics or the lore. A lot of things caused problems that were
related to the game without actually being a part of it per se. For instance:
- The restrictive GSL pushing nearly the entire third-party community away.
- Yanking PDFs (of older and current-edition) products off of DriveThruRPG (I'll admit I can't quite recall the timeline here, though).
- Issues with the DDI, from the VTT never manifesting the way it was promised (though that might be at least partially because of the tragedy that happened) to letting users sign up for one month, download everything, and then cancel their subscriptions.
- Paizo openly and deliberately offering an alternative to dissatisfied players by breathing new life into the discontinued edition (which is an offshoot of the GSL snafu on WotC's part).
- Ending the print magazines (it's iffy how much this actually mattered, though I suppose that depends on how you defined "actually," but even if they weren't financially worthwhile, the electronic alternatives struck a lot of their subscribers as being less than satisfactory replacements; this was another thing that opened the door for Paizo).
- A marketing campaign that often seemed awkward (re: the thing with the French guy) and sometimes bordered on hostile. An animation that shows your upset fans as literal trolls, who then get crapped on by a dragon, could charitably be seen as harmlessly poking fun at the controversies surrounding the rollout, but after the points listed above a lot of people weren't feeling very charitable.
That's practically a recipe for a perfect storm of controversy, all without even taking into account the things inherent to the game's rules and lore that were also points of friction. While there's no way to be sure, my guess is that if WotC hadn't made so many of those screw-ups, 4E would still be divisive for all the changes it made, but not nearly to the extent that it is today.