I was thinking of 3.5e but it seems I was misremembering how its spell resistance worked. I do like the idea of a Spell AC, and 5e has some precedent for that with how dispel magic and counterspell work. (Ability check vs a DC of 10 plus the spell’s level. In this case, you’d make an ability check vs the creature’s “spell AC” or the spell doesn’t work on them [or they only take half damage].)
Oh yeah. I forgot about that!
I think what is actually annoying me about this rule is that resistance is an actual game mechanic (half damage) but the Magic Resistance trait functions differently. It’s a bit of a misnomer. I feel like they either should have called it something else or had it apply to magic attacks as well as magic saving throws.
Disadvantage on magic attacks wouldn't really have a huge impact, as there aren't very many of those. And you still have the problem that it doesn't do anything for spells that lack attacks or saves, like
magic missile. Or have alternative systems, like 2014
sleep, or things that require ability checks, like
maze.
A universal magic resistance is hard to design when magic doesn't have universal rules. You could just have a system where creatures are immune to low level spells (like
globe of invulnerability) or have a "special save" (X or less on a d20 roll to negate a spell), or a special version of legendary resistance that only works on spells.
At the end of the day though, I found this is a lot of work to either make casters waste spell slots they're intended to use throughout the day, or make them not cast offensive spells at all, other than cantrips, which makes the rest of the party have to shoulder the extra burden and could make a fail state more likely.
Now if you're ok with all that, you could try any or all of those options, but I feel that rather than punish someone for being a spellcaster, instead, force them to be a little creative with what spells they use.
If they overrely on damage spells, throw a few resistances their way. If they keep trying to disable enemies, it's not hard to find an immunity to charm or stunning. A major foe could have a magic item like a "dimensional anchor" to prevent
banishment, or a teleportation power to escape a
forcecage.
Ideally, you let the players figure this out or at least have clues, and then they might be able to choose different spells to bring to the fight. Not every class has that option, of course, but a wand or even scrolls of different spells you'd rather they cast is still probably better than making them feel that they're wasting their time casting any spells.