I think D&D 3e had it right actually, and that is exactly why it had such a large following (and 4e seems to have split the fanbase so much). 3e gave us a game that may not have appealed as much to those who really wanted it's focus to be sharper and more narrow... but it more than made up for this in being much more capable in accommodating other play styles, add in the fact that it's the most common and well known of rpg's and you have a winner. Thus yes everyone had to tweak it to get exactly what they wanted... but everyone could find something within it that appealed to them and that they enjoyed and it was worth it because hey, everyone plays D&D. 4e has changed so that it no longer appeals to a certain portion of it's fanbase, it just isn't fun for them and it's so far away from what they want that many feel by the time they get it to that point it won't be worth it. I also feel like this is why many don't see 4e as any type of evolutionary step.
See, this is where I have to disagree. For me and my group, 3e was such a radical departure from what D&D was for us, our playstyle, and was so specific in its rules assumptions and implied setting, that it was unplayable except for a certain kind of game. We tried tweaking 3e for a number of years to play different kinds of games (Middle Earth d20, gritty low fantasy, and post-apocalyptic fantasy), and it just didn't work with the ruleset- it felt like we were fighting the system for a game that ended up being unsatifsying in play. I agree that mechanically, 4e is a different game- to me thats a good thing, but I can see how its not appealing to others. But 3e was thematically and in play a very different game than its predecessors or 4e.
IME, here is a quick list of thematic comparsons of 3e vs 4e (and previous editions):
3e:
* Good for hack n' slash dungeon crawl D&D, bad for narrative or plot-focused games. 3e did do high magic games well (which are not my favorite), but for swords & sorcery or low magic, forget it.
* Magic is king (casters were clearly the "best" characters, others were second-class). Magic also allowed a number of "I win" scenarios.
* The magic item Xmas tree was assumed in the core rules, but if you wanted to run a magic light game, it was difficult to do
* The focus on the game was rules mastery and builds, not on gameplay
* I'll get some disagreement on this last one probably, but 3e felt from level 7 or so on like a superheroes game. PCs started in 1e, 2e, and 3e very fragile and mundane (which can be fun), but by level 3 they were heroes. In fact, many 3e groups I played with STARTED PCs at level 3 to avoid the "boring levels" (and this practice was also common among many posters on these boards). After level 7 or so, PCs became ridiculously powerful in comparison to other characters and monsters in the game world, much moreso than in previous editions. This was largely due to rate of gain of power and the underlying mathematical assumptions of the system- things that were hard to modify or fix. That works fine for some types of games, but not for what we wanted to play. While 1e and 2e suffered from this to a degree, the capped hit dice, slower gain of power, and restricted spell slot advancement (no bonus spells for wizards due to high Int) diminished this considerably.
4e and previous editions:
* Good for hack n' slash dungeon crawl D&D, AND good for narrative or plot-focused games
* While powerful, magic isn't king. Other classes have their place and strengths that aren't circumvented by magic casting classes. Much reduced "I win" problems (even in 1e and 2e).
* Magic items are great and fun to have, but aren't given the weight of defining a character or his abilities. Previous editions weren't as good at this, but 4e has swung the balance towards a characters abilities, class, and powers defining their role and place in the world, rather than their items.
* Focus on game play, working with a group, and using abilities and powers effectively in game, not on building a character by picking through hundreds of options (some of which were intentionally designed to be subpar and "traps") in order to be effective. This was a major problem for four groups I played in over the 3e era. Some players were REALLY into rules, and could build insanely powerful characters, while those who didn't engage in the system mastery aspect of 3e were complete chumps compared to them.
* 4e starts heroic, and stays heroic throughout, but doesn't stray into the superheroes area from what I've seen. Starting 4e PCs are tougher and assumed to be special, but then again, monsters and NPCs are tougher as well. Rate of gain of PC power is more linear rather than exponential, and the underlying math of the system is superior and MUCH more stable throughout all levels. We've played 4e from 1st to 8th level now, as well as two short-term high level adventures (15-18th levels and 25-27th levels), and the game holds together very well and keeps a heroic tone the whole time (meaning the PCs are tough, but vulnerable with mistakes or bad tactics) without becoming bogged down and unplayable like 3e was for high-level play. The magic "I win" buttons are gone, meaning one or two round combats that were so unsatisfying in previous editions are thankfully a thing of the past. Battles and challenges now require careful thinking, skill use, teamwork, and are tense situations. I'm not saying these things were impossible with previous editions, but it was harder to maintain that kind of tone during play due to the "I win" buttons and magic items.
The whole simlationist vs. narrativist thing isn't a 3e or 4e thing- its a DM style and group playstyle thing. D&D, in whatever edition, has NEVER been strongly simuationist. If you want a strongly simlationist game, play HARN or GURPS (which are both fine games as well). D&D has always had a huge number of abstractions (hit points, AC, saves, falling damage, lack of long-term injuries, etc) meant to keep the game flowing quickly and represent a heroic sort of game, not to reflect reality in the game world. How the DM presents the world and interactions with NPCs is the main determinant of simulationism. For the record, I tend to be simulationist AND narrativist in my games, and me and my groups have not had any disconnect between 4e's rules and the outcome during gameplay.
Anyway, I realize some people will disgree with me on these points, but for a lot of us who do enjoy 4e, the things I listed are important. One system is not objectively superior to the other (and both have their flaws), but one system can be superior to the other given a group's preferences and playstyle. For us, 4e is superior and is a huge step forward over the previous edition.