All of them were driven by WotC (or TSR, in 3e's case) having already done every book that could reasonably be expected to sell well -- splatbooks for the core races and classes, the psionics book, a handful of core rulebooks for settings, and maybe a splatbook or two with new races and/or classes.
There were quite a few valid books for 3e that had been oft requested and never released. 3e didn't run out of steam or produce all the conceivable content, it was forced to end as WotC wanted to re-release the edition with DDI and grow the brand. And, if they hadn't pulled their A-talent to design 4e and were forced to rely on the B-Team and freelancers, the quality of the later 3e books would have likely been higher and the line would have done better.
It doesn't help that WotC was producing books at a staggering pace for all of 3e and the start of 4e. They killed both editions with too much content too fast.
(And PF has been doing some interesting things with the line, showing there was room for other books.)
We'd have a new edition of D&D every 3 years if players would accept it.
Never work.
Admittedly, core books sell the best and the initial rush of a new edition is a nice peak. However...
They started work on 4e back in 2005 and really ramped up work in 2006 for the 2008 release. Two years. They started work on 5e last year (2011) likely for a 2013 release. Two years again.
Now, that's two years - a full 24 months - where they're paying in-house not to write, not produce sellable content, in the hopes it will sell well. It has to be inhouse as you can't trust freelancers as much with that confidential information, so you also have to pay for the benefits and office space of said in-house staff. So pricey.
So the initial surge of sales is great, but has to pay off a tonne of debt.
What 4e did wrong was split sales. The core rules were super simple. Once you learned the basics you really didn't need a PHB. Instead, you could get any other book, such as the PHB2 or one of the two Essentials books that did the same thing or even
Martial Power or
Heroes of Shadow. You could have a group where no two players bought the same books.
However, the rule of scaling means this is killing profit. Books make profit by high sales. It's better to sell 20,000 copies of one book than 10,000 copies of two books. Because you need to pay for production costs, so the first several thousand copies are just paying for the writing and art and editing and layout. And the higher the print run the lower the cost per copy.
So you really, really want to focus sales on a single batch of core books that everyone has copies of. Everyone needs to have a
PHB. All new content should refer back to that and require it's use.
The above means if you have a long lived edition that lasts many, many years with multiple large print runs of unchanged core books, then that makes you the most money. Because you're selling books that are pure profit.
Both 3e and 4e made the mistake of undercutting the core books, the first with a massive reprint and the second initially by minimizing the need and later by producing three books that served the "core book" function.
Instead of a new edition, WotC needs to find a way to re-energize the edition every few years without changing or replacing the Core rulebooks.
Every few years, the player base will burn out from D&D. I think that hasn't exactly helped 4e, with its singular playstyle and long combats making it more prone to burnout. WotC needs to accept that after 2-3 years there will be a dip in sales as the fans go try other games and scratch the other system itch, playing all the new games that have come out over the last couple years that they haven't been able to play due to the regular D&D game. 5e has an advantage in this respect with its modularity; it's harder to burnout when the game can radically shift tone and feel.
Similarly, WotC should plan accordingly and have some awesome books on tap for a couple years down the line. They should have a dip in book production, a slow down to let people catch their breath (the perfect time for more board game and non-RPG releases) and then come back strong with a renewed edition push. A new big book of core options (that'd be the time and place for the PHB2) as well as more modules and options that really push what the system can do. After a couple of years of playing with the rules, the designers should have a better idea of where the system can bend and not break, and that'd be the time for very different rules modules and types of campaigns.
