Of course, another aspect is "does this change need to be done?"
I think we need to be careful with our selectivity of memory--notice how you don't mention the jump from 2E to 3E, which at the very least is in the same ballpark as 3.5E to 4E, maybe even a greater jump. If I remember correctly, there wasn't a huge uproar and people were generally excited about 3E (like the good Eric Noah site!), but that may largely be because of the "Dark Ages" of D&D in the late 90s. 3E was a huge boost, it re-vivified D&D in an unprecedented way and in such a way that wasn't as necessary in 2008. However, what was necessary in 2008, at least for WotC as a company, I would guess, was a major new burst in sales and, it could be argued, a cleaning up of the vast mess/wealth of 3.5E and OGL product.
1st, a "cleaning up of the vast mess/wealth of 3.5E and OGL product." by WotC really isn't possible. Any OGL product is pretty much beyond their control. If there's (or was since the d20 license was scrapped) crap out there with the d20 logo, that's WotC's own problem and directly under their control. But not the OGL stuff.
2nd, a possible interpretation of what you've said would be that part of the point of 4E was to basically invalidate a swath of the existing rulebooks/supplements. Because "cleaning up" in this particular case doesn't really mean "fixing" or "revising" it means "get rid of the crap". With the understanding of course that the label of "crap" is rather subjective.
3rd (and my main point), is that revisions and editions based on need/desire is rather different from revisions and editions done because a company needs to improve their sales.
People might have a short memory (or a convenient one), but I remember that not everything was sunshine and light when it came to the 3.0 revision. Does "It's just like a videogame!!" sound familiar? Yeah, that's from 3.0. Moving to 4.0 the main difference was that instead of people citing "Diablo" as the vidoegame, it changed to "World of Warcraft".
Regardless, it's not like the community was 100% behind a new edition. However, there were _enough_ people that felt it was warranted that it was successful. 3.5 was a much murkier revision. While there were a fair number of people that felt that fixes were necessary, there's a number of others that felt it didn't warrant the full "edition" change that happened. It also didn't feel like the community as a whole were crying out for a new edition and the loss of system mastery (something that the 3.0 designers were pretty up front about being a part of the rules) annoyed people as well.
So, from some people's perspective, system mastery is explicitly designed as a part of the rules, WotC cheerfully churns out books, and then with little warning pumps out a new edition which forces people to re-master the system (because there's lots of little changes between the two) and then buy new versions of the books they already have.
They weren't forced you say? Well, there's a pressure to do so. Stores aren't going to carry old product, they're going to dump it because a new product/edition supercedes it. 3.0 brought an awful lot of people into the hobby that insist on (mostly) Rules As Written, as well as the usual collectors and brand loyalists. And since 3.5 was the new _official_ edition, if you want to talk about the game, you've got to talk about the current edition. There are individual choices yes, but the collective pressure from bookstores, the company itself, and many fans is that you have to play the newest edition. I mean, that's what edition wars are after all... "You need to play [latest edition], it's so much better than the crappy previous one!" with "No, it's not! It sucks and all [the old one] needed was some simple [errata, houserules, fixes]!!".
Plenty of folks felt WotC lied about 4e. It doesn't actually matter if you think they did or didn't, the fact is that there's enough perception that WotC lied about 4E being in development. Star Wars Saga was viewed as a "proto" 4E by some folks and some people point to various ideas in 4E which were explored in related but earlier and different fashion in SWSE. A lot of the press I've seen about SWSE is positive.
And then 4E hit.
Now, SWSE could be seen as a "revision" of the 3.5 engine. 4E is pretty clearly a new edition. The design assumptions behind how encounters work etc is sufficiently different. And of course, while some folks might have bought SWSE, not everyone did. So while it might not feel like a "new edition" going from SWSE to 4E, there's a fair chunk of people that were looking at it from the point of 3.5 to 4E. After all, some folks are genre monkeys (fantasy), some are rule monkeys (d20 system), some are brand monkeys (WotC), and then there's the chunk that just go with whatever the primary leader is in their game group. So there's a variety of reasons for people to have not necessarily followed the rule evolution.
An awful lot of what I saw leading up to the 4E announcement was folks wanting a revision, not a new edition. SWSE could be considered to be that revision on a number of fronts, but of course it's not tuned for the dungeoncrawling that is the 800 lb gorrilla of the RPG industry.
What they got was 4E. A further step along the rules evolution path and fed back into dungeoncrawling. But quite a bit different from what some people were hoping/expecting it to be.
Oh and in addtion to 4E as a rule set, there's the cancelling of Dungeon/Dragon magazines and the whole DDI "you don't get anything unless you're a subscriber" bit added to 4E as well. And a couple of years down the road, you've got DDI which has failed to deliver on all the promises made (game table) and a new... whatever you want to call it... in the form of "Essentials".
WotC has done an awful lot of stuff over the years, that there's a segment of the population that's rather cynical. While there might be _individuals_ like Mike Mearls that are big fans of D&D and want what's best for the game, it's foolish to confuse an individual designer/playtester as being representative of the goals of the company. For example, the whole "4E won't use the OGL" thing (No, the "GSL" is not the same as the "OGL" nor is it a new version of the OGL, and this isn't the thread to discuss what the GSL really is) is viewed by the more cynical folks as a direct response to games like Mutants & Masterminds (along with plenty of others) coming to compete with WotC. the fact that Scott Rouse at one point commented that 4E had taken 6 figures to develop and why would they just give that to everyone kinda supports the idea that WotC doesn't care as much about the hobby as a hobby, but it cares about the hobby as a source of profit.
I'm not opposed to people making money, but you should _never_ make the mistake of thinking that the company has the same goal as you the GM/Player. Just like you view commercials during TV as a necessary evil, and companies view the TV programs as the necessary evil "wrappers" for their ads, you and the company might be seeing similar things, but it's from a completely different viewpoint.
On the Paizo (and other companies) side of things...
Plenty of other people have already pointed out the difference between rules/settings. So no need to belabour that.
Things like Pathfinder, Fantasy Craft, etc... they're fighting for a smaller segment of the market. While they've got many of the same considerations as WotC (including brand loyalty, system monkeys, etc) they're basically picking up the people that are dissatisfied with the already established system. So what they do is figure out _what_ it is that they want to cater to and then design for that audience.
So, where WotC tries to continue market domination of the d20-based fantasy, Mutants & Masterminds comes along and says, "I hear you dig supers and want to do some games using a system you're already familiar with. There's some differences, but... check us out." Spycraft comes along and says, "Excuse me sir, but I believe you're interested in the idea of spies and hi-tech equipment? Perhaps you should consider reading our materials" and later, "I believe that those of you that appreciate our approach to things would also be interested in exploring the fantasy genre. If such a thing is true, please consider reading these..." and Pathfinder is all, "You dudes! It blows that your game got cancelled. Check this, we dig it too, so what we're going to do is keep publishing it. I know, it's sick. On top of that, we're going to give you our own little take on it and help punch it up a bit. Check it..."
They're designing to already self-selected fans and people pre-disposed to accept their product. It doesn't mean they won't have detractors, it doesn't mean that everyone that gives their games a try will like them; it simply means that they've got a group of people _more_ inclined to give them a chance.
WotC is the 800 lb gorilla. No other company commands the market share of rpgs that they do. They've dominated the market and it doesn't look like that's going to change anytime soon. Because of the way they conduct business and the position they hold, they've painted a big ol' target on their chests. Asking why WotC gets picked on when other companies do "the same thing" is A) not accurate because other companies _aren't_ doing the same thing, and B) kinda like people complaining about Microsoft getting picked on.
And you know what? It _should_ get picked on. Any company that rises to dominance should be held accountable for its actions. It becomes a leader (whether it wants to or not) and sets an example. It's also providing a _product_ and people should demand that the product they purchase is actually up to the standards that they (the consumers) want. People complaining, people refusing to spend their money, people actively encouraging others to use different product... this is basic stuff people.
Do you really _want_ to go back to the days of AD&D/2E? I remember them pretty darn well. On the rpg front, it was slim pickings at times. And you had a company that basically did as it pleased, with little consideration for its fans. Remember TSR going bankrupt? Yeah, that came about because folks weren't spending their money. There wasn't the ability for instant feedback like there is now, and the company wouldn't have listened to it if it had been available in the first place. While there's a danger in paying too much attention to a vocal minority on the internet, WotC's (and most of its fans) push on the DDI mean that the internet _is_ a direct target market for them.