I am first off a bit disturbed by the language itself - "narrative control/options". IMO nobody, not even DM, has narrative control of a D&D game, and indeed that is, to me, a defining characteristic. The players play, the DM referees and plays with them, but NO ONE is driving the bus. That's the magic.
<snip>
If spellcasters are totally dominating the game, as a DM, I will... Hurt them...in a good way of course. If theyre scribing reams of scrolls is problematic, make it harder. Make them only scribable on a full moon, or something.
I don't want to say that these two comments are in flat-out contradiction - but there's at least a fairly high degree of tension, I think.
A more skilled player will have that fighter doing all sorts of things, regardless of what is on the character sheet.
Applying this logic, no game actually needs any rules at all. Fighters don't need defined skills? Okay. Well, maybe wizards don't need defined spells, either.
To add to Umbran's point: there are many perfectly good RPG systems out there that do as he suggests, and
don't give wizards defined spells, but rather require them to freeform their magic on the basis of give-and-take negotiation with the GM. Even 4e has elements of this with the Arcana skill, at least as it is played at my table.
But freeforming isn't the only mechanic available. And freeforming can itself be better or worse (or just differently) supported by a mechanical framework. Look at games like HeroWars/Quest, Marvel Heroic RP or 4e which support freeforming via a system of "level appropriate" DCs, with the narration then shaped to fit that; or look at Burning Wheel, which uses "objective" rather than "level appropriate" DCs but has a very strong fail-forward approach to adjudication to ensure that even if the
PC suffers when freeforming fails, the
player doesn't.
I don't think WotC (or TSR) have come out with "good" narrative-driven mechanics, and you will often have to steal such mechanics from other games.
FATE, for instance, has some very good mechanics
<snip>
We engaged in a skill challenge (it's called "dramatic conflict" or something like that in FATE)
<snip>
4e's skill challenge system supports (to some extent) the same thing, although in a social skill challenge, there's a good chance that only half the PCs can provide significant contributions.
Given the presence of skill challenges in 4e, I think your opening sentence is a little unfair (although I guess "good" is in the eye of the beholder!).
Your comment about social skill challenges is an important one. My view is that there is a need for give and take between players, system and GM here. To elaborate: the GM needs to frame challenges keeping in mind, at least roughly, what sorts of PCs his players are playing. But equally, players have to be prepared to have a go even if they will fail. Essentials tries to incentivise this by awarding skill challenge XP whether or not the PCs succeed; and DMG 2 discusses (not as well as Burning Wheel, in my view) the idea of "fail forward" adjudication.
For those players who
won't engage, the job of the GM (as I see it) is to force them. So in a combat, if the wizard or rogue is hanging back and shirking, as GM you can have a lurker suddenly enter the battlefield and engage them! Or the enemy archers open fire on them. In other words, the player doesn't get to choose to keep out of the action - the GM can frame them in by dictating the NPCs' actions. The same thing applies in a social skill challenge - the duke, or guild lord, or whomever it is the PCs are talking to
asks the dwarven fighter a question. Now the player has no choice but to engage!
To me, the distinguishing aspect of D&D is that the DM has essentially 99% control, and the players only control the 1% things that their characters control. And indeed, that's what makes it a true "role playing game",
<snip>
Giving players power outside of their characters' knowledge and abilities takes you into different territory, which can either be framed positively (as a story game) or negatively (as pure immersion-breaking metagame mechanics). In my view, that different territory is interesting for other games, but inappropriate for D&D.
I don't really get this contrast between RPGs and "story games", given that all the ostensible story games I know of - HeroWars/Quest, Burning Wheel, Dogs in the Vineyard, Marvel Heroic RP, Maelstrom Storytelling, The Dying Earth, 4e, etc - characterise themselves as RPGs, are played by RPG players, are discussed on RPG boards, etc.
If you want to play an RPG with no metagame mechanics (ie mechanics whereby players exercise power outside of their PCs' knowledge and abilities), fine - Runequest and Traveller are excellent games that I'm sure you'll enjoy! But there are plenty of use who don't mind metagame abilities in their RPGs.
And I don't see why D&D shouldn't be one such. After all, it started with a large number of them (XP, hp, saving throws).
Personally I enjoy games which give narrative control over to players... but, you have to have players that want to experience the game in such a way for it to work well... and D&D being the gateway game, I'm not sure if including such mechnics (unless presented as an optional add-on) is worthwhile or even a disereable thing for the majority of D&D's fanbase.
I woudn't pretend to speak for the majority of D&D's fanbase (actual or potential). But I don't know of any evidence that a "gateway" game is better without metagame mechanics. By all accounts new players have no trouble picking up FATE or Marvel Heroic RP.