So many interesting posts to reply to, will have to focus on just a few.
Generally liked what you had to say, but I'll quibble with one thing here: Yes, 2e produced a lot of settings - and it was successful in doing so, it sold a lot of setting books and there are still fans out their pining for the return of their favorite 2e setting! Admittedly, settings were bigger in the 90s than they are now, but it's still a viable way to sell books that don't have to bloat the game for everyone.
But 2e didn't go belly-up because it was big on settings. Rather, TSR bet heavily on selling novels, a CCG, and a collectible dice game - all of which went very badly for it. They got in financial trouble and were bought out by WotC. 2e's setting focus wasn't - as far as mere outsiders can tell based on what's been reported - what got them in trouble.
Yeah, I hear that. I'd like to think you're right that the settings had little to nothing to do with it, but I think as others have said it contributed to TSR spreading themselves a bit too thin. I mean, think of how many man-hours went into Jakandor or Spelljammer or even Birthright, and how much did they get back?
On the other hand, I dislike the conservative approach they've been taking of only re-hashing old settings. I'd like to see them be bold and create something new, even if it is only developing Nerath. I definitely think D&D needs a fresh world to explore.
I do think there is a plan:
to sale a lot of PHBs. To make a lot of money from that. And to keep selling them.
Mearls has hinted at this, a lot. That their is a steady stream of new players to tap. That you want to support the game, but not overwhelm it. Over the years, others have noted time and time again is what really sales is the RPGs main core book.
So you make a really good one. Charge a lot for it. And do enough around it to keep it viable, but that does not overwhelm new players or undermine or outdate the core book.
I'm sure that is
part of the plan, but certainly not the whole plan or even the biggest part of it. I mean, there's no way around the law of diminishing returns with regards to core rulebooks. Even if D&D taps into a whole new generation of players, I don't see PHB sales carrying the game indefinitely.
Also, the price point on the PHB makes me think they are focusing on serious gamers, trying to please the "bird in hand" with a great PHB. You have to be pretty into it to want to spend $50.
Why exactly is it an issue again if the game is overhauled and rebooted every few years? ...
You are never going to get a game that isn't rebooted. It's not because they need to sell the new shiny. It's because they want to make a better game.
Well I agree and that's part of the point of my original post. If nothing else, rebooting is necessary creatively. I mean, Roger Moore was looking pretty gnarly towards the end. It is hard to believe but even Daniel Craig will wear out his tenure and we'll need a "2020s Bond." Things change, although I suppose the definition of a "grognard" could be someone who resists change and sets up camp at a certain place and won't budge.
Here's the thing: my setting of choice is Eberron. That means that I want conversions for Warforged, Kalashtar, Changelings, Shifters, Artificers, and the Dragonmarks as soon as possible. (I don't actually need
the setting updated - just the mechanics for those things. And, I suppose, some monsters and magic items. (Seriously, I haven't even finished reading Basic and already the lack of those things is off-putting. Though not enough to stop me buying the "big three" when they're out.

)
But, equally, other players will want Dark Sun available (Muls, Thri-kreen, etc etc), others will want Dragonlance, Ravenloft, Birthright, Spelljammer... Then there's all the monsters that inevitably won't fit in the MM, each of which is no doubt
somebody's particular favourite.
So people want all these things, and WotC will want to supply those things. After all, they'd be pretty crazy not to.
But while i only want a few things, and you only want a few (different) things, and that guy over there wants another few (different) things, the cumulative result of all of these together is bloat.
But here's a question: Should WotC create a big fat hardcover edition update for every setting? Why not just create a smaller "conversion guide" - like a $20 96-page softcover? Or even just free conversions online? Or this could be where they license out older settings and let some other company do it. I'd just hate to see them spend their limited setting resources on re-inventing the wheel again and again.
I certainly hope 5e can do what no edition of D&D has done this millennium and last a full decade. I think it has a stronger chance than the previous two editions, as the system seems less complex and more stable than before. Bloat is inevitable but hopefully there's less emphasis from WotC on extreme levels of crunch and more on settings and APs.
A decade is a long time, but passes quickly these days (or maybe I'm just getting old). But I agree that 5E has a better chance. On the other hand, asking game designers not to innovate for ten years is awfully difficult. It is inevitable that new ideas come up. I'm pretty sure Mearls has a folder in his cloud called "6E ideas" or somesuch.
I like 5e but it is not perfect, I want them to eventually make my perfect D&D so I would love a 4-5 year life cycle for 5e and look forward to what can happen with a 6e already.
I started lurking at this site back when it was the best news source for 3e info, Eric's black pages how they bring back fond memories, anyway I enjoy the whole process of new editions.
I understand why some people want whatever the current edition is or their favorite edition to never end but I find the excitement of a new edition invigorates my interest in D&D.
I hope you realize that they'll never make your "perfect D&D." Or if on the off chance that they do, it won't be someone else's perfect D&D, so they'll eventually have to reboot again after that. So they'll move on from your perfect D&D...and how will you feel then? Will you still love a 4-5 year life cycle?
It reminds me of the band U2. I'm not a fan, although I appreciate much of their work. Anyhow, from what I gather they really perfected their early sound with the 1987 album
Joshua Tree. But rather than try to re-create it ad infinitum, they took a break and came back with a very different sound and the controversial 1991 album
Achtung Baby. They did something that very few huge acts pull off successfully: they remade themselves. But a lot of the old-time fans hated it, while at the same time they acquired a new fan base and, today, are one of the classic rock bands of all time.
So when people say this or that edition sucked, I say "Hey, at least they tried something different." Sort of like
Achtung Baby.