That said, there are a few things I don't (entirely) agree with in your post...But what about that spouse of a friend who might find their spouse's hundreds of rules book overwhelming? Or, very simply, what about just anyone who might want to play D&D but doesn't want to read through 800+ pages just to get started?
I don't think you are really hearing me.
You don't get friends or spouses of friends to start playing by showing the stacks of rule books, and you don't need to read through 800+ pages of rules to get started with a game. Frankly, I prefer to sit down to a new RPG without knowing any of the rules and without touching a rule book. Reading the rules probably will get in the way of my enjoyment of the game.
This is the way I play an RPG:
1) I make a proposition.
2) I interface with the rules under the guidance of the DM.
3) I recieve information back from the DM about the outcome of my proposition.
4) Go to step #1.
The important point about step #2 is that it might sometimes help that I know what I'm supposed to do to interface with the rules, but really it's enough to know that when the GM says, "Make a skill check/saving throw/whatever", I know to throw a particular dice. And even that isn't necessary. All that is really needed is the ability to take instruction from the GM about what dice to throw and modifiers to add. If the game is good, a new player will probably have been playing for a dozen sessions or more before they even bother to get a grasp of the rules, because it won't be the rules that are engaging.
Perhaps an even more important, and related, factor is the confusion with where to begin. Essentials is an attempt to clear this up and may be an improvement, but I think WotC could improve this more with the next edition, or even a revised edition, by having a clearer "product tree" that correlates with the order of publication. In other words, the starter kit should be published first, then the core rulebooks, etc, and it would be made clear that you start with the kit and then, and only then, do you buy the core books.
All irrelevant. You don't start playing an RPG by buying the books. You start playing an RPG with a GM says, "Would you like to play?"
And you don't even make new GMs by selling books. You make new GMs by players going, "You know; I think I want to GM.", or in some cases by, "Well, our GM moved to Indiana/joined the military/got married. Who wants to GM?"
Secondly, I don't think running a rules lite game means that one needs to invent rules. DM Fiat isn't as much inventing rules as making judgment calls.
Same thing, though I've no desire to argue that point again.
I work in a private high school which has a handbook with rules and policies, but we have to make judgment calls all the time that don't follow the written policies exactly and it drives some people (usually administrators) crazy.
See. Exactly my point. You are arguing here against what you just said. You have a rules light system - a handbook that doesn't cover most of the situations that come up except with vague and difficult to apply references - and so when you actually have to use the rules, you find that you spend most of the time making up new rules that there after become 'common law rules' which determine how similar future situations will be resolved. This is one of the several reasons why rules light systems are never highly successful in the RPG market place.
Finally, I will say again that it doesn't have to be either/or, either you have a complex or a simple game. It can be both: A simple core, primary game, with more complex secondary and tertiary layers. This is why I advocate a two or three layered game, with at least Basic and Advanced, but maybe also more wild optional variants like Magic of Incarnum. But the key would be making all variants interchangeable and modular and not require one to move beyond the Basic game if they are happy with it.
In this I agree. And with this, I'll introduce a really contriversial position. Really well designed RPG's aren't elegant. Elegance is a trait to be scrutinized the way you'd look at potentailly lethal microbes under the microscope. Really well designed RPG's have a vaiety of monolithic subsystems that you can ignore or add as you desire to make the game complicated or uncomplicated as you desire and in the areas you care about. Elegance tends to produce tightly coupled RPG systems where every aspect of them effects every other system, so that you can't pull on one part of it without risking the whole thing falling apart. This ends up creating an inflexible system that is complex only where the designer cared to be complex, and ignores or oversimplifies where someone else might care to have complexity, and in short which is good for only one sort of game without massive effort from a GM to make it work.
I mean, in some ways this is how most people learn the game. You start playing with what you understand and then as you encounter something new, or look a rule up, you add that knowledge or rule to your game (or not). This is how I've learned every edition of D&D, from 4E all the way back to 1E and Basic.
Ok, now we are beginning to get on the same wavelength. Yes, this is how people learn the game. In particular, this is how GMs learn the game and some of this sinks by osmosis into the rest of the players at the table. In other words, once you resolve a problem in one way, the players remember that, "In a similar situation in the past you had me roll a D20, add my dex bonus, and I succeeded with a 16. So that is 'the rule'."