I think that notions of gamism v simulationism is at the heart of the controversy that flows from each edition change.
Early D&D is designed for wargamers. In a lot of ways it is a wargame. The game's core premise involves managing time and resources and playing 'smart' to overcome obtacles and get the treasure in a hostile environment. To some extent even engaging with the combat rules at all (instead of flooding the dungeon or whatever) is a failure state. The purpose of play is to use your character as an avatar to 'beat' the dungeon (and the GM) and emerge unscathed.
By late 1st edition and 2nd edition the game has gotten very popular and its core demographic wants something different. They want a feel of playing in an epic fantasy story like Lord if the Rings, or perhaps a computer game. The game's high lethality rate is toned down (mostly by an encouragement for the GM to fudge 'inappropriate' results. Explicitly challenge- based mechanics like GP=XP are removed because they 'don't make sense'. We get skill rules, detailed game worlds, notes about the culture and habitats of each monster, and kits to give characters special rules reflecting their background or specialisation. The purpose of play is to be the third dude from the left in a LotR knock-off and to do what he would do.
Third edition represents an attempt to further head towards simulationism and 'make things make sense' by unifying the class levels, unifying thief abilities etc into the skill system, and providing mechanical representation for all sorts of trips and manoeuvres, item manufacture, monster skills, character options, prestige classes within the world, etc. However, this is married to a deluge of exception-based character options designed to allow you to 'build' a powerful character and thus overcome the game's challenges through player skill. There are even deliberately underpowered 'trap' options in the feat list for canny players to avoid. 3e is thus much less coherent than previous editions, it is simply trying to achieve two contradictory aims at once. Is play about challenge, or about exploration and verisimilitude? Should I build my character to be the most effective, or the most interesting and setting-appropriate? You can see that this dichotomy was what broke the 3e playtesting, with WotC by all accounts running the new game under the old assumptions and therefore missing the reality of how things actually work. In the real world playgroups adapt to this incoherence by either outright shifting back to challenge-based play as driven by the new rules, or by trying to ignore the parts of the rules that drive things that way through a mixture of fudging and selective cultivation accompanied by admonitions not to be a munchkin or a power-gamer.
Cue 4th edition. 4th edition says look, you all clearly want the challenge-based game the previous rules tried to provide, so let's provide lots more structure on what a fair encounter looks like, give you lots of cool AND BALANCED options and manoeuvres to choose from, and get rid of the OP spells, restrictions, and assumptions ('baggage') that was getting in the way. Let's also be really crystal clear about what we're trying to do and give you unambiguous advice about how to make this game sing. Verisimilitude can go f itself, right? This is just a game.
Cue shock and outrage from half of the audience. Cue also genuine fun and excitement from the other half of the audience - at last, a D&D that lived up to the experience it had always promised!
Cue 5th edition. As the market leader we can't afford to lose half of our audience, even if the other half is having the time of their lives. We need to soften the directness of the rules, throw a lot more ambiguity into our language and GM advice, and add back in all the old assumptions, trappings, and inconsistencies as clearly these are the things people like about D&D. We need to appeal to the widest-possible audience again by making our game sort-of palatable to both sides. We'll even play some vague lip service to narrativist ideas by putting in some stuff about player goals, inspiration, etc.
And there you go, most successful edition ever, just enough of each agenda to satisfy both sides (but delight neither). I think that commercially-speaking GNS-incoherence is probably the correct course of action for WotC, as much as it disappoints me from a game design point if view. As the market leader they want to appeal to the widest audience possible, and as the genre prototype, with an extant history of incoherence and ambiguity, your audience is already pre-disposed to selectively ignoring and interpreting your books to get what they want. 'Of course you have to fudge, how else can you keep the characters alive until the climax of my pre-planned story!'
I think the maths on this is very different for non-D&D games, which are already reactions to D&D in different directions and are therefore chosen (or will be chosen) based on how well they 'fix' D&D's 'flaws' and move strongly in any given direction and towards a particular creative agenda.
