But It is slightly already this way. Death spells only work on the living. Drop a vampire enemy and all the poison, stunning, death, disease, paralysis, and sleep spells don't work. Drop a troll, they are forced to use acid and fire.
What you're describing falls under the topic of research and kill, which is a different D&D trope. If you're familiar with the creature, or the PCs do their homework, you know not to use those methods when confronting the creature. It's the same as researching the dragon you're hunting, so you don't bring a dozen fireballs to fight an ancient red.
Those elements are there to add versimilitude first and foremost. They're not typically there to balance an encounter against broken magic. Trolls can be put down using a torch or vial of acid, more easily than with a spell. Is there honestly any balance to be found in needing to use lightning bolt instead of fireball against a red dragon? I would say no.
You don't need to add versimilitude to spells in the same way. Is it honestly any more realistic to have teleportation magic with less than a mile of deviation limited to epic level? No, even teleportation magic that always deviates by at least 10 miles, or a magic system where teleportation is impossible, is still internally consistent.
Most BBEGs, pencil and paper or video game, use the "I'm bigger and badder so that doesn't work." trick. I don't see the problem with harder than normal challenges countering certain tactics and strategies.
Some games do, others don't. The later 4e solos often have reactions to powerful attacks like stun, or means to mitigate the impact such as having multiple turns in a round; I can't think of any that are simply immune. I've always found it lame when a BBEG says "Haha, that doesn't work on me just because I'm the BBEG". That's an indication of weak design, to me.
A red dragon is immune to fire because he's a fire dragon. The BBEG is death warded essentially because the DM is metagaming. There's a difference.
A lot of DMs will justify it as the BBEG being smart. Firstly, not every BBEG should be a mega genius with access to limitless magical resources. Secondly, it's the equivalent of the pope spending all of his time in the popemobile, even when he's inside the Vatican. Sure, it might make sense that he's in the popemobile when he's traveling, but it'd be a bit laughable if you walked in on him meeting with the Cardinals and he's standing there in the popemobile. But that's essentially how BBEGs behave because otherwise PCs will use game breaking magic to wipe them from existence without breaking a sweat.
You can have a fun game without giving the players nuclear options. In fact, I think nuclear options lessen the fun. Most players will use tactical nukes if you give them tactical nukes, but is it really all that satisfying to win by means of an auto-win button? My group eschewed save-or-die in 3e because, the first time we used it, it ruined a battle that we were looking forward to. We never used it again.
Besides, it causes the game to devolve into spell-counter-counter... Instead of walking into the BBEG's throne room, casting Death Spell, and laughing about how pathetic the BBEG was, the cleric casts Dispel Magic first (to remove Death Ward) and
then the wizard casts Death Spell (cue laughter). So now the next BBEG needs a spell or magic item to make his Death Ward more difficult to dispel. So the PCs go and find a spell that allows them to penetrate that, resulting in a spiral into absurdity of counters and counter-counters.
The problem is D&D's magic is powerful. The weakness of magic is its strictness. Magic did specific things, could only be used some many times a day, had to be performed perfectly, had just as available counters, and there were availability issues to start with.
It wasn't the easy way. The experience was just as great.... when it worked.
The problem (largely) isn't that D&D's magic is powerful. It's that the limits are in the wrong places.
A teleport spell that allows you to instantly teleport between continents is undeniably powerful, but if it never can do better than a mile of precision, it suddenly becomes useless for scry-buff-teleport. It can still save you months of potentially dangerous travels if you're journeying to distant lands.
A death spell with an hp limit is quite powerful. You can look at any low level creature and it will die. However, if the hp limit is 50 while the BBEG has 200 hp, the PCs will have to engage him in combat before finishing him off with Death spell. Heck, you can even have the Death spell deal a small amount of damage if the BBEG is above the hp limit, as it causes his heart to skip a beat but he powers through. Essentially, the Death spell transforms from an anti-climactic opener, to a satisfying finisher.
Scry could be tweaked to be able to look into the past and the near future, always seeking out a pertinent moment, however the visions might also be disjointed and hard to understand. That's powerful, as you'll never scry the BBEG doing absolutely nothing of interest. You might get a hint of his plans, or some insight into his personal motivations. However, it's absolutely useless for scry-and-die tactics because what you're seeing may have already taken place (or may be yet to come).
As you can see, all of the above are still quite powerful. However, they've been limited with regard to how significantly they can break the game. In all of the cases, the BBEG no longer has to take measures to ensure that the PCs can't use these abilities. This makes BBEGs who do take such measures special. If the PCs find that the BBEG has found a means to prevent others from Scrying him, they may wonder what it is that he's afraid they'll learn. Could it be something from his past? Or perhaps it is something he's destined to accomplish? This may motivate the PCs to research the BBEG using non-magical means.
This is just what I came up with now, off the top of my head. Professional designers with a year or two to work on the issue could almost certainly do much better than this (IMO).
