've just noticed the huge contrast between that, and D&D. The differences between 4E and 1st Edition AD&D are so massive, it's like one game isn't even recognizable as a version of the other. Interesting. I wonder why that is?
Two words: grognard capture.
It's really interesting to take some RPGs and look at their editions, from 1st to their current ones. Most RPGs follow a similar track:
1. They were introduced with relatively digestible, 64 to 128 page or so rulebooks.
2. Each successive edition incorporates more and more rules found in expansions into the core.
3. Eventually, the game grows into a bloated mass that only the hardest of the hard core play.
In essence, the game began its life in a state that allowed it to capture a ton of fans. Yet, the presence of that body of fans encourages the publisher to move away from a model that let it capture them in the first place. Instead, it now focuses on pleasing the group it captured. In so doing, it erects barriers to gaining new gamers.
It's hard to avoid that trap. The people playing your game have already consumed that original, 64 page game. They don't want it again. They want that game and all the cool expansions you released. After all, that's what they're playing with now. They don't want to go back to the basics.
D&D managed to avoid this. 2e actually cut back on options and sought to simplify the rules. 3e rebuilt the core mechanic. 4e refined 3e, simplifying the core and creating a more manageable series of options.
Rather than expand the core of the game, D&D has focused on improving its core. Now, other games do that too, but in general their developers placed a significantly higher premium on compatibility. After all, that's what their current fans asked for.
OTOH, the folks behind each new edition of D&D have had faith in their ability to overcome the fears of incompatibility with a promise of superior mechanics. So far, that gamble has paid off.
I think that D&D leans too heavily on sale in bookstores, and sales to new gamers, to every give them short shrift compared to existing players. Ideally, you keep both groups happy, but you can't sacrifice the D&D crowd of 5 to 10 years from now to keep the hardcore, extreme edge of today's crowd happy.
My personal belief is that publishers radically overrate compatibility. I think that while there are lots of loud complaints about it, a sufficiently interesting and improved design not only helps bring in new players but also re-energizes your existing fan base.
Publishers make the mistake, IMO, of trying to sell only to the people who are buying the current edition of their game. There are also lapsed players, people who moved on to other games or who just grew bored with the old one. A fresh new take not only keeps your game accessible, it also pulls in players who left your game for one reason or another.
I also think that, in many ways, D&D's position as the market's behemoth makes it easier for gamers to accept changes to it. It's pretty easy to find rants about how terrible D&D is. Gamers who are plugged into the culture around the hobby, regardless of how they feel about the game, can rattle off a litany of typical complaints about it. D&D is big enough that everyone talks about it.
For smaller games, only the people who like it are really engaged by it. You don't see consistent criticisms. I think that's actually a significant drawback for them. It's nice to have fans who love your game, but the people who seethe and rant can be pretty useful when it comes time to figure out what you need to fix. If all you hear is praise, you're stuck with guessing.