Can't say as I agree with the idea that emulating the story is what matters.
Ok. Let's keep that in mind though when we look at the rest of what you say.
First, if 'emulating the story' isn't what matters, it isn't clear from what you write what you are actually emulating.
I play RPGs mainly to imagine being a different person, with a different personality and capabilities, facing different challenges. If I'm playing a Middle-Earth wizard...
Ok, stop there. What does that mean, "I'm playing a Middle-Earth wizard...", if not, "I'm playing a Wizard like those stories in 'Fantasy Novel X' about Middle-Earth." What meaning is there in saying, "I'm a Middle-Earth wizard.", if the resulting story doesn't appear to be like those set in Middle-Earth? Is it by that point simply name dropping? Is it enough to be set in Middle-Earth for you to use Middle-Earth like names, even if the characters and situations are nothing like those of actual cannonical Middle-Earth stories?
'm going to be making decisions about whether to use magic in a given situation. I want that decision-making process to be a matter of getting inside my wizard's head and trying to see the world as he sees it, think about it as he thinks about it. The process of reaching a decision is as important as the decision reached; if the process involves thinking "Well, I've only got one fireball memorized today," when the imaginary world contains nothing that even approximates such a constraint, I'm not a happy gamer--even if the end result, in terms of what I do when, looks a lot like what Gandalf did in the books.
I find that a really strange position. You are happier with what you imagine the constraints of Gandalf to be even when the constraints you have imagined don't result in a story like the one Gandalf was in, than you would be with constraints that aren't what you imagined them to be when they result in the outcome of a story like the one Gandalf is in? I mean, it seems trivially true to me that the story is what we can concretely relate to the story, and the constraints are mere artificial inventions we create for the game. Why care so much what the game rules are if they result in the story we are aiming for?
I think what you are missing here is that in no novel is the mechanics of the world as systemized as they are in game. There really is no underlying system behind the scenes and events of Harry Potter or The Lord of the Rings. Since there are no actual rules to the system, anything you come up with will have to have features necessary to the game which are not found in the text. The description of the genera conventions regarding anything, but here particularly magic, will be thinner in the story than they will be out of the story. For example, in the story Storm Troopers have pin point accurate blaster fire, but out of the story in the game we must come up with some convention which says in effect, 'except when shooting at heroes', which is not a constraint actually in the story. If we don't do that, then we will be baffled to find that the game drawn from how the story describes itself doesn't actually when put into practice resemble the story.
I agree that naive implementations of the magic systems in fantasy books and movies often result in PCs being far more aggressive with magic use than their "canon" counterparts. However, that's not because implementing such systems is impossible; it's simply because the designers didn't think hard enough about the constraint side of the equation.
I think you are wrong. I don't think it is merely a niave implementation that results in that. I think its that simply the story is not systematic, so any system which only has those constraints which are drawn from the story will not in fact be reversable to produce the story from the game. To create a system which produces the story from the game inevitably requires doing something more than what the story itself describes.
Before you can design Gandalf's magic system, you have to think about why Gandalf doesn't go around flinging fireballs all the time. If it's a personal/moral choice... well, why should a PC be constrained that way? My character isn't Gandalf! Maybe I want to play Saruman instead.
But Saruman doesn't go around flinging fireballs all the time either. The problem you have here is that Tolkien is also doing story emulation, and the use of magic in myth is not systematic either.
And if I do want to play Gandalf and restrict my use of power, I can do that without the rules mandating it.
In that case, why do you need rules at all? That's sort of the player version of, "The game system has no problems if the DM changes the rules as needed."
If it's a practical choice--the Valar have imposed restrictions on him, or too-aggressive use of magic will enable Sauron to locate him and nail him to the ground, or there's a corrupting "dark side" effect from abuse of power, or some other reason--then write an appropriate limiting factor into the rules, and get your local munchkin to bang on it a bit and make sure it holds up.
Because all of those things are wrong. They don't actually reflect what is in the story any more than Vancian magic does. The Valar commanded him to help the people of Middle Earth, but they don't actually have a switch where they can override what he does. Using his power is left up to his judgment. It was Gandalf's decision to veil his power and majesty. Gandalf can use magic all the time...
he just doesn't want to. More over, there isn't a corrupting "dark side" effect from using his power. Gandalf's 'magic' since it has its source in Gandalf is inherently good. Gandalf doesn't suffer corruption as a result of excercising his native power. Gandalf suffers corruption for the same sorts of reasons anyone is corrupted - bad motives, wrong actions, evil thoughts, etc. Whether or not he's actually using his 'magical' power isn't really the issue. The connection of his power to corruption is indirect. Saruman gets corrupted, but there is no indication that it was because he used his 'magic' more often or more vainly than Gandalf did. Heck, Gandalf uses his magic to make fireworks, and he's mostly commonly using it to skulk about Middle Earth like a rogue.
And the problems of attempting to emulate the mechanics of magic from the text are virtually limitless. We get almost no information about how magic works in Middle Earth, or what the limits of Gandalf's power actually are, or what range of spells he can perform, or really anything we need to make a system. All we can say is in the story he did this and that, and look at some system and say, "Within this system, can we do this and that (and only this and that and no more)." And guess what, make Gandalf a 6th level Wizard and not only can we do this and that, but only this and that and no more.
And that gets us much closer to a good Middle Earth simulator than any of the suggestions you gave.
Sure, D&D probably isn't perfect for M-E, but right off the bat it beats anything you just came up with.
Finally, when you say "when the imaginary world contains nothing that even approximates such a constraint", I wonder how you know. There is nothing in the text that indicates what actual mechanical constraints Gandalf's spell casting is under. We don't have a book of Middle Earth magical lore. For all we know, 4e D&D perfectly captures the actual mechanical constraints of spellcasting in Middle Earth. Or maybe 1st edition AD&D. Or maybe GURPS. Or maybe Ars Magica. Or maybe something else. We know that Gandalf doesn't cast spells all the time, that he can be fatigued and apparantly unable to cast spells, and that there are apparantly limits to what he can do, that these spells can involve verbal incantations, and that he has a certain repertoire effects - some of which he uses on multiple occassions. That's all we actually know. So pretty much any system that has wizards that don't cast spells all the time, where wizards can run out of spells, where they have limits on what they can do, where the spells can involve verbal incantations, and where they have a limited repertoire of effects can handle the the story. The details of those mechanics are hidden to the observer of the story and are fundamentally irrelevant, just as the details of Gandalf's magical practice are hidden to the reader because they are fundamentally irrelevant.
The only way we have of judging the mechanics is whether they emulate the story. If they fail to emulate the story despite being like you imagine the 'real' mechanics within the story, then they aren't the correct mechanics. If on the other hand, they do emulate the story then they are at minimum 'good enough', which is in most cases the best we can assert since the underlying 'real' mechanics don't exist.
Are we talking any version of D&D or RPGs in general? Are we assuming a lack of rules for a given scenario means the game as a whole doesn't handle the corresponding story element well? Are you equating setting with story or are you being all inclusive? What is the litmus test for "handling well"? I'm not trying to be pedantic, I genuinely ask cause it's an interesting (and polarizing) question and I want to know where you're coming from. That aside... My off the cuff answer is no version of D&D can handle Ursula LeGuin's Earthsea setting's magic conceits well. Magic is rare but usually always available to wizards albeit dangerous to wield, possible to exceed your "break limit", and the gift of magic can be lost. AFAIK there aren't rules for any of that in D&D where spells are safe/reliable, you cannot exceed your spellcasting level limits, and you can't lose the gift of magic.
just saying its not that odd when you step out of D&D. Other than retro and d20 clones I don't know of another game that uses spell slots 
