But we do ourselves a disservice to describe that with shabby marketing terms like 'system mastery' which makes it sound like that is a feature instead of a bug. I'm glad to see that 'system mastery' which was a term that WotC seems to have invented to excuse itself for deliberately printing combinations of deliberately overcosted trash filler cards and deliberately undercosted 'chase rares' in Magic the Gathering as a way of extorting the player base into buying more of their product has gotten the negative reputation that it deserves, but I don't actually believe that WotC deliberately built 'system mastery' into 3.0 because it makes no sense for them to have done so.
IIRC, the first time I encountered the term "system mastery" was in a post by Monte Cook saying that they deliberately built it into the game. They did this to try to maintain interest amongst the 'gearhead' players - they would remain fascinated by the possibilities of finding the "perfect" optimised character.
Let's call it what it is, poor balance. I don't think that it was ever intended that toughness was supposed to be a poor option, and I hope it was never intended that power attack was to be so dominating.
Again, Toughness was listed as an example of an intentionally poor feat, both as something for the 'gearheads' to learn to avoid, and also as a "feat tax" for really good Prestige Classes. Power Attack is a somewhat different case - it only really became over-powered with the 3.5e revision, when 2-handed weapons got that massive boost. That one is poor balance.
To claim that 'system mastery' was the root of the problem
It's not the root of the problem, or even really a symptom. What it does, though, is make it very difficult for new players to sit at the same table with old hands. Either the old hands will deliberately nerf their characters (and likely be frustrated at doing so), or they will end up with much more powerful characters than the new players, likely leading to those new players becoming frustrated with the game and leaving. A game that leaves half of a mixed group of players frustrated really isn't a good thing, IMO.
If people really wanted a 'rules light' system, then we'd all be playing True20 or something even more simple. Quite obviously, 80% of us, weren't and aren't.
Three things:
1) None of us are immune to marketing. None of those other games you cited are D&D.
2) There is Ryan Dancey's old argument of "Network Externalities" to consider. If I'm looking for a new group to join a game, I have to play the game they're playing. If 80% of players are playing D&D, there's an 80% chance I'm going to end up playing D&D. If I'm looking to put together a new group, I'm going to want to use a system that the majority are familiar with. If 80% of gamers are playing D&D, that game is going to be D&D. The very dominance of D&D means that it will continue to dominate.
3) 80% of tabletop RPG gamers is still a tiny number. We have no way of knowing how many potential gamers have taken a look in Borders (or equivalent), seen a big row of book after book, seen the core rulebook gift set at 832 pages of text, and immediately decided that they would much rather go play WoW instead.
However, you are absolutely right: sometimes, what people say they want and what they will actually pay for and use are two entirely different things.
In fact, I can't really think of a 'Rules Light' system that was ever anything but a niche market.
Name one RPG other than D&D that was
ever anything other than a niche market. The closest is probably Vampire at it's height. And "Vampire: the Masquerade" was considerably much more rules-light than D&D.
Heck, D&D itself is pretty niche.
And if you actually starting taking away subsystems, you'll get strong backlash against it because a plurality will say 'that wasn't the complexity I thought you meant'.
True. As we've seen with 4e and the parsing out of the core, though, you may get a lot of people complaining, but an awful lot of people will keep buying anyway. As you yourself pointed out: what people say and what they do are not always the same.
And while you will lose some people if you lose complexity/options, this may well be worth it if you bring in more new players to compensate. Sure, some of the people here would have a disproportionate effect, being the hardcore who buy all the books... but some of those new people will represent a "new hardcore", and pick up the slack.
A good example of juggling complexity is M&M. M&M's particular take on this is to make character creation require a PhD or a spreadsheet (or both), but to make the game resolution mechanics themselves simple in the vast majority of situations.
That sounds like a good trade-off, but for new players it is exactly backwards. Require a new player to read even one rulebook, use complex math, use complex tools, or make complex choices, all before starting to play, and you won't have a new player: you'll have yet one more person who's decided to play WoW instead. I have literally seen the enthusiasm die in the eyes of potential players when they saw the (3.0e) core rulebooks. "Do we have to read all of that?"
Get people playing quickly, and the games will sell themselves. Even if the systems in play are actually very complex, that's mostly fine - they player can learn those as they go along. And, in fact, character management beyond initial creation can also be quite complex, since by that point they'll have some investment in the game. But RPGs
must get people playing as quickly as possible - and if that can be done with their own custom character, and without restricting them to a 'beginner' class, so much the better.