I did not generate a different magical system, per se, but areas of my world where the "laws of magic" function differently.
The greatest/largest area where this is so is my setting's desert empire. The desert was created by a huge cataclysmic influx of arcane and divine energies being manifested in tandem. As such, while I haven't had to use that are much in play (odd how adventurers will do just about anything
not to travel through desert! haha), my concept was that the desert in general and a large section deep within was essentially "wild magic" or, maybe the term nowadays would be "chaos magic".
When creating this, I deemed that any arcane spell casting occurring within the desert suffered an automatic 50% chance of "alteration" (included in the random chart are several percentages of "nothing happens", along with various other effects of various detriment). Wizards native to the desert empire/tribes are immune to this effect (either due to a natural immunization -i.e. a mutation of their ancestors' presence when the cataclysm occurred- or decades of working with the uncertain magic). However, since magic works differently outside the desert, they assume a 25% chance of spell "malfunction" if they are outside the desert or another "wild magic" area.
Now, 50% doesn't sound like much. But for a spellcaster from outside the desert, who is used to their spells doing what they're supposed to, having things go awry half the time, with no telling
if or
how they might, is a definite handicap.
[EDIT] The 50% is mitigated by caster and spell level. -5% per caster level over 5th. +5% per spell level over 2nd. So a 8th level wizard trying to cast a 3rd level spell would have a 40% (50 -15% for caster level. +5% for 3rd level spell) chance of spell mishap. That same mage casting a 1st level spell would have only a 35% chance of malfunction.
Obviously, this has led to only the most powerful non-Thelitian mages (unless they are laden with reliable magic items) to dare to adventure within. Considering there are only a handful of mages exceeding, say, 10th or 12th level in my world, it is a rare and still dangerous thing to find a non-Thelitian wizard traversing the desert empire.
[/EDIT]
Items work just fine, essentially reasoning that magic items are "processors" of the energies they require to function...or, in the case of magical/enchanted weapons, "charged items", and the like, the fact that they are/have their own "stable battery" of arcane energy to work with. So a wizard, from outside Thelitia, with a few useful items is not completely without resources...just sorely bereft of reliable spellcasting.
Druidic magic is all but useless. The Great Spell essentially annihilated the "natural/Green magic" of the world in that area.
While, again, I haven't had to use it in play, I believe I have a note around somewhere in my setting material that 1st and 2nd level spells for a period of days equal to the druid's level are available...before they exhaust their personal reserve of natural energies.
3rd and higher spells, I reasoned, require some "drawing"/harnessing on Orea's natural energies...which the desert does not have. 1st and 2nd level spells can be summoned "at will" by druids in my game [EDIT] (i.e. they do not need to "memorize" those spells at the beginning of the day, but can cast as many/as needed as their daily allotment allows) [/EDIT]. So consider it using up the druid's own personal reserve of power before spells simply do not function.
Divine casting works as normal (lending to the rise of the clerical sects as an extremely powerful political force within the desert empire).
I have also used other smaller areas of magical alteration: places where, perhaps, Necromancy and Conjuration work fine but other types suffer random effects or don't work at all, Divination "blind" spots, etc... that are the result of some ancient curse or placed by some fabled wizard or powerful entity, places of large magical outpouring/battles/alteration in the past, whatever. These, however, are usually quite localized (a single "dungeon" or castle or even just a single floor or room within a stronghold, etc.)
So, I guess the point is, there are ways to make magic "dangerous" or "unpredictable" without having to generate an entire new system of magic working.
Another way of working this in, as a world-wide phenomenon, would be an outlining of the concept of "ley lines" across your setting. The closer to the lines/points of power, the better/more magic works for the caster. The further away, a mage might be hard-pressed/exhausted just by casting a 1st level spell, but completely unable to summon enough power to cast a 3rd or 4th...or some such.
Again, this goes more to setting-fluff/flavor than system/"crunch." But just throwing it out there as an option.
Fun topic/thread, though!
Have fun and happy [random] casting.
--Steel Dragons