TSR cancelled the Red Box.
Seriously, the (perceived) buy-in is just too great.
Take 4e: The three core rulebooks run to 832 pages and have an RRP of $105. In order to play the game, each prospective player has to be familiar with large chunks of at least the "Player's Handbook", and the prospective DM has to have read large chunks of all three. Then, each player has to create a character, made up of large numbers of complex options (many of which a new player simply will not understand), and the DM has to spend some amount of time putting together an adventure (or spend yet more money).
Or consider Pathfinder: The core rulebook and the bestiary together run to 880 pages or very dense text. Indeed, there may be as much as 50% more to core Pathfinder than to 'core' D&D 4e. And, again, there's the reading, the character creation, and the adventure generation.
Look at other RPGs, and you'll generally see much the same things over and over again: Vampire has two required books, one of which is very thick. Warhammer 3e has an RRP of $100. Shadowrun 4e is a thick book of complex rules and options. And so it goes.
Or, alternately, there's the Starter Set. A low-cost, easy-to-approach introduction to the game. Perfect, no?
Well, no.
Firstly, WotC made the mistake of releasing the Starter Set some months after the core rulebooks. Consequently, it usually gets displayed as one of 20-30 D&D products on a shelf, and doesn't exactly stand out from the crowd. Indeed, you might be forgiven for not even realising it exists. (Plus, a big shelf of endless supplements is every bit as intimidating as looking at the 832-page rulebooks that you 'have' to read before you get to play. ("You mean I have to buy all of that?"))
Secondly, the Starter Set doesn't have character creation rules. Thus, the game immediately loses one of its great selling points - the ability to play your own custom character.
Thirdly, Starter Sets past have not used the 'real' rules of the game, but instead a stripped down subset. (I don't know if this is true of the existing Starter Set.) So, when you graduate to the 'real' game, things are subtly different.
Fourth, and finally, the Starter Set becomes essentially worthless once you graduate to the 'real' rules. So, if you don't like the game, you've wasted $20 on the Starter Set, and if you do like the game, you've wasted $20 on the Starter Set.
(The reason I mention the "Red Box" above was that that was the last time the Starter Set seemed to be 'done right'. Although, even there it fell foul of using the wrong rules, since it was for BECM D&D and the 'real' game was AD&D.)
How to fix this?
Actually, I don't think the buy-in issue can be fixed with 4e, any more than it could be with 3e. The product line would require a fairly significant rethink, that almost demands a new edition to put into practice. Here's how I would do it:
Firstly, build the game around a single Core Rulebook, no more than 250 pages (fewer would be better). This would include a basic set of options (say four races and four classes, levels for the first tier, and basic sets of powers, magic items and monsters). Crucially, this would be the same Core Rulebook that is used both in the Starter Set and the 'real' game.
On release day, release two products in parallel. The main product, called simply "Dungeons & Dragons", would be a big boxed set aimed at new players. It should include the Core Rulebook, a quick-start guide, pre-generated characters, a book of pre-generated adventures (each suitable to be run in 3 hours or less with minimal set-up required), character sheets, dice, dungeon tiles, miniatures, and anything else that you might need to play. Heck, it should even include pencils!
The second product would be the Core Rulebook, sold separately. This would be aimed at converts to the new edition.
Moving onwards, I would support the Dungeons & Dragons set with additional expansions over time, each adding another book of adventures, more pre-generated characters (maybe), more dungeon tiles, more miniatures, and so forth. Maybe one or two of these per year, in a nice shiny box of goodies.
For the more traditional crowd, I would support the game with three circles of supplements. The first circle would be the "expanded core". This consist of between 3 and 6 additional books, and would put back all the 'missing' options that we're used to from our core game. Crucially, these are supplements to the Core Rulebook, not replacements, but they would also be designed on the assumption that you're using them together or not at all.
The second circle would consist of the supplements we're used to - splatbooks for the races and classes, pre-generated adventures, and so forth. These will also assume you're using the expanded core, but will also (mostly) work with just the Core Rulebook - they should serve to intrigue the core player to investigate the expanded core.
And the third circle would consist of things like the settings, perhaps also the DDI, and so on. Again, these would assume the use of the expanded core, but again they should mostly work with just the Core Rulebook. I think WotC have actually handled the settings lines mostly right with 4e - I would be inclined to support a setting a year, each with an intentionally limited run. (Perhaps a Setting Guide, a Player's Guide, maybe a Bestiary, and a couple of adventures.)
Regarding the Rules:
Character creation in 3e, 4e and Pathfinder is too damn complex. Players have to roll and then assign six stats (or use point buy, which is even more complex), choose from seven races, choose from eight classes, and then choose three different types of powers, each from their own unique lists, choose skills, choose feats, choose equipment, choose a name... why am I not playing WoW again?
(And, especially in 3e, Heaven help you if you make the wrong choices! You're a half-orc bard, are you? Cool, enjoy playing a character that doesn't work quite right for the next several months!)
So, I would strongly advocate bringing back the Basic tier (though not the ultra-fragile characters - 4e is right in eliminating these), and also making the choices less difficult up-front. Surprisingly, 4d6-drop-lowest in order may be the best 'default' method of attribute generation. The choice of race should probably be mostly cosmetic, at least at first (racial feats, powers or talent trees can expand this later). Each class should probably provide a fixed set of iconic powers for the first few levels, with options opening later. (And it should go without saying that any halfway-decent set of attributes, race and class should be at least reasonably effective. 3e, in particular, has too many rubbish options available to the unwary.)
Feats should be deferred to second level, while skills should probably be a matter of picking the ones you have (as in SWSE), rather than assigning ranks. I'm not sure what can be done about equipment - perhaps each class should include a list of half a dozen quick choices, each of three or four options (choose scale mail or chain mail; choose a longsword and shield, or a greatsword, or two shortswords...).
For all those experienced players who are reading this dramatic reduction in options and swearing they would never play such a game, provide sidebars offering the ability to customise the character even at low levels (just not as the default), and also advice for starting the game at the first level in the Expert tier. (The goal is not to strip out options for the sake of it - the goal is to make it easy for new players to get playing their own custom character without spending an hour on doing so.)
For the rest of the rules, a similar mentality should prevail: simplify!
Part of this will be forced on the design by only having 250 pages to work with, at least initially. Things like special combat maneuvers should be deferred to the expanded core. Stacking rules should be really obvious (perhaps even going so far as "you can only have one temporary effect on you at a time"). Conditional modifiers (and especially trivial conditional modifiers) should be reigned in or eliminated. Oh, and do away with "system mastery".
And so it goes.
Reduce the buy-in, getting people playing quickly, ensure your "Starter Set" retains its value, and then provide the options desired by the hardcore later.