I think what you describe is ONE option, not the only one. Ask anyone who ever played a caster in Dragonquest, it is VERY VERY possible to make a simple, even dull and boring, and incredibly dangerous, magic system.
The question is more like "why would anyone run a PC who risks death every time he exercises his ability?" If magic is really hazardous enough to SEEM hazardous, then it is pretty much a given that it will have to ACTUALLY be pretty hazardous, at least over the longer term.
I think that's all true. But the problem with that approach is not one I think a game from 1980 would have particularly concerned itself with - balance. Imagine we have a simple system, and we give the player an option of two classes - warrior and mage. The warrior is 50% more potent than the mage, but the mage has a power where he can be twice as potent as the warrior only there is a 1 in 6 chance he'll die. The two classes are not balanced, and tables will evolve to one of two positions - characters aren't disposable so always play a warrior, or else characters are disposable so always play a mage. IMO, games balanced on random chances of death where the player has no input in what happens just aren't balanced and don't really present any interesting choices. Making the math more complex just obfuscates that problem.
(Incidentally, I have the same problem with many complex critical hit/fumble systems that produce random, "You behead yourself with your sword." results. They take control out of the hands of the player and guarantee in the long run no matter what they choose to do, they lose.)
You could introduce that sort of thing into D&D and the same sort of logic would follow. If wizards and warriors were roughly balanced to begin with, why would you now ever play the wizard? Or if they weren't balanced, then wizard play would evolve toward the most potent actions they could take which didn't involve significant risk, along with the option to 'nova' with a high risk maneuver when the odds strongly favored defeat anyway. And this still wouldn't feel balanced, nor would I think it feel "dangerous".
We've seen magic that is dangerous in D&D, it's just never the province of the player in traditional play. Gygax loaded up his writing with dangerous magic. He just silo'd it off into magical traps and special magical tricks that tended to have immediate obvious consequences. Gygaxian dungeons are loaded up with dangerous magic that can make for play as paranoid as any Call of Cthulhu investigator opening up a musty tome or peering into some dark space. Or think about the Deck of Many Things or the artifacts from the 1e DMG. I think it is a critical difference that the player doesn't know what is going to happen, not even in a broad sense. Gygaxian special magic can do anything, and I think Gygax understood the importance of keeping that safe and secret when he wrote the 1e artifacts with fill in the blank powers selected secretly by the DM.
Now, a COOLER system might be one that really let you trade off effectiveness for danger in interesting ways. That would almost surely start to get into your 'DM workload' though.
Not necessarily. The 'DM Workload' part comes from the fact that the player doesn't have perfect information. He's carrying for example several magic items with strange quirks that work in special circumstances or are triggered by unusual things. If the DM can remember all that, the player is left in a world were strange things happen and they are not entirely sure why or what it means. But remembering all of that is difficult. Likewise, you can make spells work like unidentified magic items with clauses and features and side effects that the players don't know about, and which might horrify them if they did. But then you have the burden of storing all those rules texts and resolving them secretly. That halts play. You are almost going to have to certainly flip to the appropriate entry, whereas by contrast it is a table rule in my game you can't cast a spell unless you have the rules in front of you, so spellcasters generally do that flipping on other players turns so they can be ready to go.
Still, it would at least have the virtue that it would be a choice as to how much this was engaged.
Absolutely.
a character could ping around with 'at will' type level powers and be modestly effective, without invoking any of the real big danger/mystery part of things. Or they could dig down and do crazy stuff, making dire promises, sacrificing souls, whatever, in order to have a more serious impact. Some of that could involve "well, if you fail that Arcana check then the demon is going to break loose..." etc.
Truly dangerous magic causes the demon to break loose and you are not necessarily immediately the wiser that it has. It involves the recognition that lighting the candle with the flame might be giving yourself cancer or causes a hag to snatch a baby in the nearest village or that your eyeball might mutate and sprout from the tip of a tentacle. You can do that with random effects, but that ultimately isn't that interesting, because there is no choice in the system. Dangerous magic gives the illusion of control. You think you can figure it out and use it, but there is always this nagging part that thinks, "What don't I know? Is that weird red glow a fairly harmless side effect, or am I really going to regret this in the long run?"
I still do that sort of stuff, but I have to confine to a couple of instances per party and Gygaxian style weird locations, because its otherwise just too much to keep track of.