But that boom was just that, a boom. It didn't result in more and more young kids getting into comics as the years went by. Instead, you had this big bubble, the first generation to come of age with comic book stores, and as that generation aged the publishers keep chasing them. Trying to get lapsed readers back, trying to find new ways to get current readers to buy more.
Of course they keep chasing that same core audience! The thing is, in order to keep D&D going at all, they need a certain level of sales to be maintained. It is far, far easier to get those sales from the existing customer base than it is to draw in large numbers of new players.
The fundamental problem, though, is that once a group has the core books for the system, they never need to buy anything more in order to game forever. This was true with BECMID&D (as soon as I got my Expert Set, I never needed anything else), 1st Edition (PHB, DMG, MM), 2nd Edition (same), 3e (same), and 3.5e (same). It's true of Pathfinder (Core Rulebook + Bestiary).
It's a bit less true of 4e, but only because the perception has become that D&D must include Barbarians and Monks and Bards, and is "incomplete" without these elements. Even so, once you buy the 10 Essentials products (or even some subset of these - some aren't really essential), you need never again spend any money. One
could choose to play 4e forever with just the PHB, DMG and MM... though I suspect it would start to feel very limited quite quickly.
I think that WotC (and Paizo, and ENWorld for that matter) have actually found the solution to this already - subscriptions. Whether it's to the DDI, or the monthly Adventure Path, or a Community Supporter account, if they can get enough people to sign up on a monthly basis, they can basically become assured of getting that "certain level of sales", and have a guaranteed and predictable revenue stream, thus securing the future of the line and freeing them to be a bit more creative.
So what we need, WotC, Mike Mearls, is Conan D&D back. The market needs both, because the Adult Gamer, like the Adult Reader, wants different things from his hobby than the teenager. Even when that teenager is 34.
I fundamentally disagree with your assertion that Conan = kids, Skywalker = adult. They're just different tastes. I also disagree with your call for two different games. In fact, having a 'kids' version of the game would guarantee it would never get a sale, as the kids would all want to play the 'adult' version.
As for your statement that 4e is the best designed game ever... well, that's true only for certain definitions of 'best'. It is, frankly, utterly unsuited to my preferred mode of play. (But, see, even that's just a statement of taste!)
We have now Essentials and I think the mistake WotC is making is that they think they need to make the game *simpler* to appeal to new players. Well, maybe that would help, I'm not sure. Certainly AD&D did well in that demo and it was full of weird rules.
I think you're badly wrong here. I have long felt that D&D sets up a huge barrier to new players getting into the game. The buy-in is just too great. A new player coming to the game cold is faced with the purchase of three core rulebooks ($105 RRP), then 832 pages of reading (sure, the player only needs to read part of the PHB, but
someone needs to read all the rules), then the generation of an adventure (2+ hours of work), then the generation of a bunch of characters (40 minutes each, being generous). And
then they finally get to start having fun.
It's no wonder the game is in trouble!
Now, of course, this is mitigated somewhat by in-game demos. It is mitigated somewhat by the use of pregenerated adventures (at yet more expense), and pregenerated characters, and this, that, and the next thing.
But I can get a WoW subscription for a fraction of the cost (most of us already have a PC that can run it, especially those who might be potential gamers), and I can get up and playing in, what, an hour?
So, 4e was a step in the right direction, by simplifying the rules and going exception-based (much as I personally dislike it). Essentials is another step in the right direction, especially in bringing in an easy entry route with the new Red Box. But the game still needs to go further.
Now that doesn't mean I think we have to go for simplicity at all costs, and as others have mentioned, going rules-lite doesn't actually help the novice DM control the game. But I do think WotC need to
dramatically lower that first hurdle to entry.
(FWIW, the new Red Box
still suffers from one of the biggest problems that has plagued the entry set since the cancellation of the
old Red Box. If you like the game, you play it for a few weeks then have to throw it away and buy in to the "real" game, thus wasting the money you spent. Of course, if you don't like it, you've wasted the money. We need an equivalent of the old "Blue Box" Expert Set. And if the current rules structure can't support that (and it probably can't), then the current rules structure needs changed.)