Having a chance to fumble is a debatable balancing factor. Let me flip the question back to you:
If fighters could do ability score damage (instantly defeating creatures with an ability score of 1 or 2, irrespective of their hit points) but ran a risk of horrible fumbles, would you consider that balanced?
Personally, I would answer no, but that's more of a stylistic preference than an objective truth. Though, that is contingent upon the risks being truly detrimental. Which can itself be an issue. Having a 5% chance for insta-death just for using your abilities isn't fun. Having a 5% chance of having your head replaced with a tentacle may actually be desirable for some players, while for others it might completely ruin their character concept along with the desire to play that character.
I concur that it would be undesirable for fighters to work that way--though balance-wise I think it could, in principle, be made functional.
For casters, however, there is a long tradition of catastrophic spell failure in fiction and in roleplaying--Warhammer Fantasy Roleplay, Dungeon Crawl Classics, Lots of OSR adjacent rulesets (Low Fantasy Gaming comes to mind). For this reason, the idea appeals to me--you're right, though, it is an acquired taste. I think it
could be balanced, though that balance might be difficult to arrive at.
Casters have the greatest flexibility in exploiting the system. They have hundreds of potential abilities to choose from (spells) which can target any ability score. A player who keeps this in mind can work the system in their favor and is likely to be quite powerful.
There are few non-casters with similar flexibility, though the battle master comes to mind as a subclass that might be able to effectively exploit the system. Other subclasses won't even notice the change, like the champion fighter who neither benefits nor suffers as a result of this change, since they have no save abilities. Then there are classes like monk who have a core ability that depends upon a saving throw (stunning strike) and they are likely to be very swingy under such a system. Against low Con monsters they won't be able to miss a stun. Against high Con monsters they won't be able to land a stun.
Converting saves to NADs is the more balanced way to go, IMO. Sure, the lines on it aren't anywhere as elegant, but there's a reliable engine under the hood, and IME that's what matters more when you're actually playing.
Okay, point taken. The
math issues with departing from bounded accuracy carry a lot more weight than the gain in
simplicity. I'll have to ponder it a bit.