Here's the issue as I see it.
You have spellcasters, who use magic. Magic can basically do anything. In fact, if your magic system doesn't allow magic to be used to do anything, it's typically seen as incomplete.
Here's the thing though, what if magic can do anything...
eventually. Part of the problem from earlier editions to 3e is that spellcasting has become automatic with a reduced chance of interruption. That problem was compounded by most players not seeking to take advantage of the ways how you could disrupt casting. 4e did a great job of separating out rituals from "spells" but then didn't capitalize on it. I sincerely hope that D&Dn continues with this distinction. If rituals are powerful but take time, resources and planning, and spells are cast but are easier to disrupt, then I think you start bringing back the combat power level of casters to their more mundane colleagues.
Then you have non-spellcasters, who use things that are not magic. Sure, they might have a handful of magic items, but we're already talking about a magic item system that is largely beyond the players' control, so they are not to be relied upon. Non-spellcasters can really only do such things as are believable (in some cases, perhaps, Ripley's Believe-It-Or-Not believable, but believable all the same) in the real world.
Why can't non-spellcasters be good with the resources they are good at: loads of hit points to avoid being affected, better defenses that increase in a mundane way (skill bonuses to AC) rather than capping out and relying on magic to take them the rest of the way? Why not use the bulk of stuff from 4e that IS believable, fun and makes mundane characters interesting to play as well as poweful?
The problem is that these two groups have to interact. Each group has to be able to challenge (and potentially overcome) the other in the normal course of play.
This is no problem for spellcasters. Magic can do anything, including turning a spellcaster into a melee monster, or just outright killing a non-spellcaster with an offensive spell.
Make the melee-monster-magic a ritual that takes an hour and uses resources. Or you could take it further and make it usable only from a scroll, where a scroll is not a spell or ritual impressed upon vellum but is its own individual thing, unreplicable as either spell or ritual.
You need to learn how to make this scroll and you don't make it easy. It might take years of study, expensive resources or be incredibly specialized requiring a whole bunch of pre-requisites or all of these. Crafting the scroll takes time and it might not be 100% effective all the time, it might have side effects. A caster using it might not be certain of what they get, that is part of what I was referring to as far as magic being dark and mysterious.
A swordsman can be confident with his weaponry and his capacity with that weaponry. A wizard might have confidence with certain magic but would be pushing themselves with other magic. Spells don't need to be deadly, they just need to be capable of fizzing, or using further resources of the spellcaster whether it be their capacity or aspects of their health. You make it so that a player cannot spam these special abilities willy-nilly. They only pull them out when necessary rather than be their typical modus operandi.
This is a problem for non-spellcasters. Remember, magic can do anything. A magic system that does not allow you to fly, or turn invisible, will be seen as incomplete by a lot of players. And that flying, that turning-invisible, those things are very, very hard to compete with from a non-spellcaster's perspective.
It does not have to be. What if the introductory spell for flying takes a standard action to sustain meaning a spellcaster can only use move/minor actions and thus the spells they can cast becomes limited to spells castable as minor or move actions? Such spells are obviously less powerful. Or you make Flying a ritual that takes time to cast but only has a short duration. There are so many ways you can limit these spells.
As for the mundane fighter, if he can put an arrow into the caster, perhaps the caster needs to use all of their actions to avoid plummeting? Make flying riskier so the caster needs to rely on circumstances being favourable or being made favourable.
Invisibility might not be 100% perfect. When an invisible character moves, the light might refract around them in a perceptible pattern that perceptive characters have a greater chance to notice.
What I'm getting at here is that you put limits on these spells so that there is an innate fairness to the magic; a fairness that admittedly wasn't necessarily there in previous editions.
It's really a sort of Venn diagram issue - the sphere of control over the world that spellcasters enjoy is complete (or so near to complete as to be indistinguishable from it during the normal course of play), while the sphere of control over the world that non-spellcasters enjoy must be limited or players begin to decry a lack of verisimilitude.
Why can't spellcasters be good at what they are good at which is perhaps a small subset of the "complete" set of magic. Perhaps there is a further subset of magic that they can cast but is uncertain, leaving the rest of the set of magic beyond their reach. Sometimes what they do will be effective but the guarantee of its effectiveness is no longer there.
Yes, sure, magic is limited in terms of use. You only have so many spells per day. The fighter can swing his sword forever. That's awesome. But that's not how the assumed course of play in D&D has really ever worked. Spellcasters ration their spells, or supplement them with cheap consumables, and are thus able to extend their effectiveness.
There are so many ways to limit magic as I've highlighted above. Rationing their use through the Vancian system is really just a small part of what is possible and richly thematic.
But above all, the non-spellcasters are essentially tethered to the effectiveness of their spellcasting companions; the hit in party effectiveness that a party completely out of spells takes is too great to risk continued adventure, and we get the X-minute adventuring day problem.
Extend the capacity for at will spell casting. Also reduce the PCs capacity to control their environment at higher levels. Making shifting/teleporting or MMMing expensive and difficult rather than adventuring macro number one will go a long way to forcing a group to soldier on (rather than the DM having to constantly rehash the "
x" is going to happen in "
y" hours time trope).
So for those who need verisimilitude to be a priority: How do you reconcile a personal need for verisimilitude with a professed desire for magic to be powerful and flexible, and a professed desire for non-spellcasters to be viable, exciting, and useful contributing members of the party?
You reduce the access and automaticity (making up a word) of spellcasting. It is powerful given optimum conditions, but those optimum conditions are not easily garnered, particularly during combat. Combat needs to be a rough place for the wizard to be.
You want the villain to be capable of flying, and turning invisible, and protecting-from-arrows himself, because that's the sort of magic D&D ought to have. But where does that leave the non-spellcaster, confined to the bounds of the believable? How does he compete?
Protection from arrows in 3e was too powerful. There needs to be a "lesser" version that is like a 4e shield spell but requires actions/resources from the caster to use while the effect lasts. Have the full version be a higher spell that is good against grunts but not as effective against the dedicated archer.
Casters in 3e really had carte blanche in their actions. The thing is, it is not difficult to ratchet this back a step; limiting such magic but still keeping the wondrous essence so that the caster is still "special" enough. You can limit the caster without resorting to the "hp damage plus condition" ethos that turned a significant subset of wizard-lovers off of 4e.
Again I say, it should not be difficult to keep caster-players happy while keeping mundane characters relevant, necessary, powerful, desirable-to-play and significant.
Excellent discussion by the way.
Best Regards
Herremann the Wise