I'm not sure that's 100% true though. 2e and 1e saw a split that, at least anecdotally, was just as wide as 3e to 4e. I've seen estimates as high as 50% of groups not making the switch over. And there are many, many examples of people talking about how they skipped over 2e to start again with 3e.
That's not much different than the 3.0 v. 3.5 situation. The thing is, 3.0 and 3.5 were similar enough to continue using most sourcebooks and modules, and they were nearly 100% compatible in fluff (I don't like gnomes with banjos, but I survived). And options were left open to make either game your own. 1e v. 2e was very similar; you actually cannot tell the statblocks apart in many cases, and even most PCs can be converted with a pencil and an eraser (bards being the exception). 3.0/3.5 did not split the fanbase, although it undountedly derailed sales and ghetto-ized certain rules discussions.
3e to 4e is much like 2e to 3e with one big exception; the vast majority happily converted from 2e to 3e. Some people playing AD&D never converted to 3e, but they were probably never convertible. Going from 3e to 4, there are a lot of holdouts, and even the people who buy into 4e are not necessarily setting 3e aside.
Some people blame Pathfinder, but that is rewriting history. As soon as 4e was announced, people started cranking out alternative logos for 3e third party support. Paizo announced their project pretty early, first of all in order to capture a lot of playtest data, but I think secondarily to scare anyone else out of the field who might be considering a somewhat updated rules-system rather than a straight continuation. Paizo or no, the 3e engine was going to keep running. If someone was willing to publish Labyrinth Lord, you can bet someone would happily port the 3.0 and 3.5 rules over.
In other words, Pathfinder was "sufficiently D&D," perhaps even sufficiently 3e, to win a large segment of 3e holdouts which 4e was unable to attract. It seems a no-brainer that WotC could be more successful with any product than a third party publisher would be. It is my hunch that 4e became popular ONLY by virtue of it being branded D&D, otherwise it would be in the Runequest/Warhammer 3e niche. It follows then that if WotC had published, say, Pathfinder, that alternate universe version of Pathfinder would be even more popular than Pathfinder is now, probably a near complete capture of the 3e market. Some people obviously were burned out on 3e but not enough to tank the line, particularly if you offered a Pathfinder-esque reboot.
The so-called "edition treadmill" is a good thing. I never balked at, every few years, picking up some errata, some minor changes, and oh yes, expansions. Those near-compatible versions are a good chance to clean house, just as 3.5 got rid of the Weapon Master and assorted other things. I was happy to buy Pathfinder; new art (even though it's not precisely my favoriate style), refreshed mechanics, and a chance to review all that has gone before.
Why was 4e less popular than it could have been? Well, let's see, your target markets are:
1. people who liked 3e before but don't like it as much now, and would welcome a game that was NOT very similar to what they've been playing, and
2. people new to D&D
I think WotC figured #2 was where the money was, but you know what? 4e is too bulky and arcane. And someone who isn't playing D&D now may not be likely to do so in the future.