Legendary resistance is a cheap hack, jammed into 5e because we didn't have a better solution to the broken control spells that we had to include in the game for tradition's sake.
How's that for an intro?
As incendiary as the statement might be, it's fundamentally true. D&D changed over the years, but its content remained the same. The spells that give DMs headaches today had counters in AD&D when they were first released. As the game shifted over time, those spells retained their core functionality while monsters lost their defenses against them. What's a designer to do? First, let's take a step back into how things used to work.
Resisting Magic
Roughly speaking, magic resistance was a flat percentage chance that a spell failed to function when it targeted a creature. Powerful creatures, particularly those from other planes, had magic resistance. Let's take the bone devil as a point of comparison.
In 5e, the bone devil is CR 9 and has a Wisdom saving throw of +6. Not bad! At 5,000 XP, we can expect it to face characters from 6th to 13th level. It probably has a save DC against PC spells that ranges from 15 to 18 or so, meaning it saves about 60% to 45% of the time. It also has magic resistance - the 5e version that gives advantage on saves - so it succeeds around 75% of the time on average.
In AD&D, the bone devil has 40% magic resistance. Target it with any spell, and there is a 40% chance that the spell simply fails to do anything. Its save against spells succeeded on an 11+ on a d20 roll. That means between magic resistance and a static save that it succeeds on a save about 70% of the time.
That's pretty close! Except there's one problem. 5e gives characters a LOT of ways to mess with saving throws. 5.5 in particular is rife with them. In AD&D, characters have few if any ways to alter a monster's saves.
On top of that, in the typical AD&D party only one character - the magic-user - had access to spells that could shut down a creature. The cleric could do it, but their offensive spells were weaker. The fighter and thief, barring access to a magic item, relied only on damage output. In contrast, the default in 5e is that every character can expend resources to shut down a monster.
If the party focus fires a creature and correctly understands that control effects are key to winning, they can shut down a creature through sheer volume of saves. That simply wasn't an option in AD&D.
In context, the math between 5e and AD&D lines up well. In practice, an AD&D creature saves maybe once per round, while a 5e creature might need to save at least once on every character's turn.
"Save me!" Cried the Mighty Balrog
This design approach leads to bad tension in the game. The DM wants a creature to have a fighting chance. The players want to turn its action economy to dust. Legendary resistance is bogus because the PHB gives the players those options. What fun is there in giving someone toys and then taking them away?
Let's start with determining the scope of the problem. I don't care if chump monsters fall victim to spells or special attacks. They appear in large numbers, so taking out one or a few doesn't do anything.
Legendary actions are a giant, blinking, neon sign that says, "I'm a boss monster!" So, let's add this mechanic to replace legendary resistance:
Legendary Defense: When this creature makes a saving throw, it can expend a legendary action to gain a +5 bonus to all saving throws until its next turn.
I think this works as a get out of jail free card. A spell or effect slows down a monster, but doesn't grind it to a halt. It then provides a model for some unique effects, say for a mindflayer.
Spiteful Mind Lord: When this creature makes a saving throw, it can expend a legendary action to gain a +5 bonus to all saving throws until its next turn. If it uses this ability in response to an Intelligence, Wisdom, or Charisma save, the creature that created the effect takes 15 psychic damage.
There! Now we have some texture that helps shape how to deal with monsters. Here's another one for a devil. In this case, our boss monster devil struck a contract with some allies to help it out when it is in a pinch.
Diabolic Bond: When this creature makes a saving throw, it can expend a legendary action to gain a +5 bonus to all saving throws until its next turn. The first time it uses this ability and fails its saving throw, 1d3 barbed devils teleport to open locations within 30 feet of it.
How about stuff like
forcecage? In theory the concentration mechanic is supposed to keep those in line. What if we did something like this to get PCs to burn through more spells:
Twisted Weave: At the end of this creature's turn, each active spell that affects this creature or an area within 100 feet of it is polluted by its corruption of the weave. Creatures of its choice that cast those spells take 25 force damage per spell they cast.
You could model that with different damage or introduce other effects to mess with PCs.
How about dismissal or similar magic?
Walker of Hidden Ways: If an effect would teleport this creature or send it to another plane, this creature instead teleports up to 500 feet to an open space of its choice.
Putting it All Together
As you can see, there are plenty of ways we can counter the metagame presented by the PHB. The trick is being aware of what the characters can do and how we as DMs can counter them without simply saying no. Monsters need tools to counter the effects they see in play, the ones from the PHB, rather than potential hazards they might face.