And I have to agree with Jack7 that it was the increasing complexity of rules that appealed to the geeks and nerds and supported increased popularity of D&D in particular as a roleplaying game.
Don't get me wrong, I have nothing against complexity. Indeed, a game that is too simplistic will very quickly lose interest.
However, there's a place for complexity, and increasing complexity. And, IMO, the
core rules is not that place. Get people in the door with a nice, easy learning curve, and then ramp up the complexity for those who want it later.
In fact, I would argue that the Core Rules for the game should provide the simplest 'full' experience of the game. So, provide a limited but representative range of classes, a limited but representative range of races (or even investigate race-as-class in the core), and a limited but representative range of levels (and spells, monsters and magic items). But do so in a straightforward way, without cluttering the rules with a hundred and one different options (4e), or a hideous mess of mathematics and builds (3e).
Then you can use your supplements to expand the game in all three ways: by adding higher levels to the game (and the inherent complexity that comes with those), by adding a greater range of customisation to the existing classes/races/etc ("X Power" in 4e), and by adding whole new modules (power sources) to the game for those who want those.
That way, everyone wins - the people coming into the game have an easy learning curve, the people who crave complexity get that, and
it is easy to transition from the one to the other as you wish. But providing a "real game" that is a 1,000 page mess of options, and then a "Starter Set" that is both a feeble pay-for preview and, especially, that uses a dumbed down ruleset will fail - the 1,000 pages is too much for the curious newbie, but the crippled "Starter Set" will just drive them to ignore it and jump to the "real game" (and promptly abandon the whole thing).
(It's also perhaps worth noting that what was once true may no longer hold. The same nerds who were once attracted to D&D for its complexity can now find the same in other places, such as WoW and Magic, both of which provide easy ways to jump in and more manageable learning curves. Meanwhile, D&D itself is trying to go mainstream, and attracting a mainstream audience probably means that the "complexity that attracts nerds" may well not be the way to go anyway.)
Even if 1E AD&D (for example) was hopelessly inefficient in presentation, needlessly and even confusingly wordy, had incomprehensible mechanics it still succeeded wildly despite any such handicaps.
I wonder, though, just how many people
actually entered as a result of those rules.
It seems to me that a large number may well have transitioned to 1st Ed from the older editions (or the Red Box), and so already 'knew' the game before coming to those books. Indeed, they may well have not even read the 1st Ed books, but instead just played the game they knew with the new classes/monsters/spells. In effect, playing OD&D or BD&D but with 1st Ed trappings.
A second group probably came to the game largely through word of mouth (the same way almost nobody ever reads the rules for "Monopoly" - it's almost always taught by someone else... who probably hasn't actually read the rules). That is, they were playing a game that was recognisable as D&D... but may well not have actually reflected the game in the books particularly closely.
And then there's a third group that probably came about because of local "experts". The
DM read the rules, and was passionate about the game, and drew the players in. But the players knew just enough to play - they may well never have read the 1st Ed PHB, never mind the DMG. (And bear in mind that the 1st Ed PHB was
much less involved than current offerings. It included the races, classes and spells, but it did not include such things as combat or exploration rules, or even character creation!)
So, while 1st Edition was wildly sucessful, I'm really not convinced that this was because of the
books - I'm inclined to think there were other factors at work, and that emulating the books themselves is therefore a flawed strategy.