A variant that used to be seen a lot during the 70's, but I never see now is 'klutz factor' systems - a %age chance of spell failure, increasing with each recasting. If the spell fails there is a same %age chance of a backfire.
FWIW I considered allowing a metamagic klutz factor system, whereby metamagic could be applied on the fly, but there was a 10% chance per missed spell level that it failed, and a like chance of backfire.
e.g. attempting to extend a spell on the fly with no increase in spell level was a 10% chance of failure (followed by a 10% chance of backfire). Attempting to quicken and maximise a spell on the fly would be a 70% chance of failure (followed by a 70% chance of backfire). Depending upon the campaign a backfire could range from opposite spell results, to target the wrong target, to damage like a scroll backfire, to have a demon appear to claim your soul. It was quite configurable
My favourite alternate magic system is probably
Call of Cthulhu d20 - I'd love to run a fantasy adventure where that was the magic (learn each spell individually, casting a spell costs you sanity and/or ability damage).
I rather liked the flavour of the Sovereign Stone d20 magic though, especially with the concept of excess heat generated by magic use meaning that most magic users wear skimpy clothes under a big robe that can be removed quickly when you need to cast!
Cheers