It might help to analyze what went into each D&D edition change...
The split of D&D into AD&D and BD&D was driven by differences between it's creators...
AD&D 2e was driven by a desire to clean up AD&D. Gygax had been ousted by then.
D&D 3e was driven by a desire to make the brand profitable to it's new owners. ...
This doesn't seem quite accurate to me, or rather what you say is true but you emphasize rather universal motives for specific editions. I mean,
every edition was motivated by "the desire to make the brand profitable" and "a desire to clean up" the previous edition, at least to some degree.
In that sense, I would argue that each edition was motivated by at least two factors: profit and game evolution. Each edition might have specific, even unique, motives behind it (as you point out with AD&D/BX), but those two are part of every edition, every book really.
I would say this was the real clean up. Very much driven by a desire to make AD&D work as a game even as the game was brought back to its roots...back in the dungeon.
Right. I still see 3E as the most substantial change in D&D's history. I'm not just talking about mechanics, but total impact: 3E ushered in a new era, quite literally "21st century D&D."
The “answer”? Lots of experimentation at the margin, but true edition reboots only when needed and with real value added that reflects both lessons learned and new stuff that actually works. I don’t think any edition has quite got there. Still too much group think, designer preference, and condescension towards the fans, and yes, the problem of “we need money so lets make core books cause they sell the best”. However I do think that the game has in general gotten better, inspite of all this. It has evolved.
I agree. In my opinion 3E took two steps forward but no steps back, whereas 4E took two steps forward again but one step back, which is where all the controversy comes from. An example of this is powers - they provide a high degree of flexibility, modularity, and consistency, but they also are problematic in terms of homogenizing classes (at least initially) and making 4E seem too "video gamey."
Yet despite its flaws, 4E provides a stronger foundation for further game evolution than 3E did. Why? Very simply: DDI. DDI provides a tremendous opportunity to evolve the game because it will always be up-to-date, always include errata, and always remain compatible with the current iteration of the game. This allows immense flexibility and leeway in terms of introducing new ideas, new sub-systems, new Incarnum-esque projects (now whether or not WotC capitalizes on this potential remains to be seen, as its potential lies more in
creative rather than
financial fruit).
D&D does many things, and because of this, can never be "streamlined" like CoC.
Hmm...maybe you are right. But I think this has more to do with the sheer number of supplements and players that D&D has, as opposed to CoC. How many people have played and currently play D&D? The rumor is 25 million and 1+ million, respectively. CoC? Let's say a hundred thousand and ten thousand. The sheer number of D&D players makes the amount of tinkering that much greater.
The amount of supplements, and the continual creation of new material--whether published or house rules--gives D&D a feeling of continual change and development. People are always coming up with new things; in fact, that is part of the game, the culture of D&D that was encouraged from the beginning.
I do not get what you are saying regarding the focus problem. 4e for example is very focused as some kind of minis game and its tactics.
Right now, but that is 4E and might not be 5E, at least not in the same way. D&D as a whole is not so focused, except on the most basic premise: fantasy adventure with lots of monsters, treasure, exploration, etc.
Furthermore, for most gamers, D&D is
the RPG from which everything else is an evening's diversion. A game group might be in the mood for something different and play Call of Cthulhu or Traveler or Reign for a few months, but they will invariably come back to D&D. Sure, there are diehard "indie gamers" that won't touch D&D, but they are a minority. To put it another way, most gamers play only D&D, and of those that don't only play D&D, most of them
mainly play D&D.
Why is this relevant? Because when a D&D player plays CoC, it has a sense of impermanence to it; you might play it for a story arc (until you go mad!) and then go back to D&D, even the same campaign you've been playing for years (decades, in some cases). So it is more like a mood piece, something you want to try out and enjoy for what it is, but not fully enter into for an open-ended period of time. The sheer amount of time that people spend on D&D increases the probability that tinkering arises, and it also increases the chance of "edition boredom," the readiness to try something new that is still D&D.