I think "overcasting" has its roles in literature and some very limited game-play circumstances, but attempting to wrap rules around it attracts the wrong kind of powergaming. I think it is best implemented on a case-by-case basis, and with no expectation that the rules will be consistent.
If I were designing such a system, though, I'd hit the mages where it hurts - lose permanent maximum HP (plus an equal number of current HP to keep the totals synchronized)! If max HP go to 0, permanent death and possible soul destruction. Perhaps the mage gets permanently sickened a la Raistlin if he goes below a certain threshold. How many max HP for various "overcasts"? I'll leave that as an exercise for the reader (a term used in textbooks for "I'm too lazy to work out the details").
This system wouldn't hurt clerics as bad, but I suppose the system could still work for them too... at least this would be better than Divine Metamagic!