the reasons for D&D being the way it is serve story, genre convention and gameplay as much as (if not moreso) than logic.
See, to me, DnD has nothing to do with serving genre convention or story. Then again, I don't game to make up stories. Either as a DM or as a player. To me, any story that comes out of the game after play is a bonus, but is certainly not the goal of any game. The goal of a game is to play the game, not to expound on my personal abilities (or lack thereof) to write or create a story.
In other words, to me, the players combined with the DM create the story. If the genre conventions are conflicting with the rules of the game, then one of them has to give way. I never game with the mindset that I am trying to recreate a particular story. I game to explore the aspects of a particular character thrust into a particular situation. Genre convention is meaningless to me.
In other words, I couldn't care less how Tolkein, Howard, or R. L. Stein would play the game. I only care how I play. If the setting completely ignores aspects of the rules in the service of some sort of genre convention, then, well, I want no part of that. The point of the game is to service the game.
Again, as I said before, if you don't want your players to do certain things, change the rules so they can't. Don't leave the rules as is and then whine when the players don't follow along. Complaining that DnD is high magic while allowing all character classes is pointless. The second you have a wizard and a cleric in the party, your game is high magic. Every encounter will feature magic.
If you want to get around that, then you have to change the RAW. There's absolutely nothing wrong with that. Thieves World, Conan, and other systems do EXACTLY that. That's the point. In order to recreate those genres, you MUST change the RAW.
However, if you play RAW, don't expect the game to emulate any single genre. That's silly. RAW is meant to encompass many genres; so many that it is more or less genre neutral. Or, rather, DnD has become a separate genre unto itself, distinct from most literary genre.
While I completely agree that there needs to be suspension of disbelief, my mind shouldn't be forced to be so open that it falls out my left ear. Six year olds should not be able to poke holes in the setting within the first ten minutes. Granted, everyone has a different tollerance for this sort of thing so YMMV and all that. However, I find the idea of completely ignoring what I see as very, very simple concepts and sweeping them under the carpet as almost insulting. Yes, be willing to swallow some of the lies in the setting - that's cool. But, don't expect me to see the setting and never question any elements ever.