As a basic concentration slot system, how about:
Number of concentration slots = caster level + 3. (As a starting point - I haven't really crunched the maths but I think this may be too generous at high levels. Perhaps 2/3 caster level + 3 might be a bit better)
A spell generally requires a number of concentration slots equal to its level divided by 3: 1st-3rd level spells require 1 slot, 4th-6th level spells require 2, 7th-9th level spells need 3. To cast an instantaneous spell, the caster must have enough empty concentration slots at the time of casting. Increases to spell level due to metamagic etc increase the number of slots required proportionally. So a quickened maximised scorching ray would require 3 slots. Spells that currently have a 'concentration' duration require double the normal slots. Cantrips do not require concentration.
(Alternative: casters get caster level + 3 concentration slots, and spells require level/2 slots to be cast. Again, I'd like to emphasise I haven't done the maths on this one...)
Assigning a concentration slot is a free action performed as an integral part of spellcasting. Reclaiming concentration slots happens automatically once a spell is dismissed, dispelled or has its duration expire.
Some spells are much more demanding, and require more concentration slots. This is a balancing issue, and should be used to shape play. I would personally put teleportation spells into this category. This would nerf buff-scry-teleport tactics to come degree, since the concentration slots required to perform the teleport would limit available buffs. Combat spells that use ridiculously expensive material components as a balancing factor would also be a candidate here (hi, Forcecage...), although we're still stuck with expensive material components in non-combat spells like Raise Dead, since in the situations when these are cast, concentration is less likely to be an issue.
Other uses for concentration slots:
- as a free action, can devote a free concentration slot to give a +2 bonus on dispel checks, either to dispel an enemy's spell or to prevent the dispelling of one of your own.
- as a free action, devote a free concentration slot to give a +2 bonus on Concentration checks to avoid fizzling a spell due to distraction (taking damage mid-casting or environmental conditions etc).
- perhaps some way of using concentration slots for the purposes of counterspelling? Shouldn't steal the thunder of Dispel Magic from the abjuration specialists, but allowing some measure of spontaneous counterspelling from all casters can only help increase the 'mageduel' feel that D&D has never done well. And making counterspelling easier and more accessible can only reduce the power of casters in general, and so improve things for non-casters at high level.
This system would also allow the restructuring of save-or-die spells from their 3.xe swinginess or their Pathfinder just-another-way-of-losing-hit-points-iness. Save or die spells would now come into effect at the end of a duration - perhaps three rounds for the lower level ones (Slay Living), or one for the higher level ones (Implosion, Destruction). The caster has to maintain concentration on the spell for the entirety of this duration, or it has no effect. This gives the target the chance to ward himself against the spell effect somehow, get the hell out of range, kill/distract the caster, compel the caster to drop the spell in order to deal with tactical developments elsewhere, etc, etc. There may be penalties accruing over the course of the spell, or there may not be - depends on the individual spell.
Of course there's scope for all sorts of new feats/spells, just like there is every time you introduce a new subsystem. Feats to get new concentration slots, or let your familar contribute. Debuff spells that tie up a casters concentration slots or make their spells more draining to maintain. Etc etc. Perhaps antimagic could be reworked to make spells *hard* to cast rather than completely impossible? Depending on the caster level of the antimagic effect, all spells could require X more concentration slots than usual, which would mess up low-level casters completely and limit high-level casters to their lesser spells.
General goal here is to limit uberbuffing as a viable tactic, and make casters less nova and more measured (thus making combat last longer and giving non-casters more chance to shine at high levels). Caster can still teleport in and go boom, but he's going to be a fair bit more circumspect since he'll have fewer buffs up, and be a lot more vulnerable if his big metamagicked boom doesn't do the job first time. Also, making counterspelling, dispelling etc a bigger part of the game.
Big area i'm leaving uncovered here is what to do about creatures with spell-like abilities. Do they need concentration slots, if so, how many and how are they balanced? Needs thought...