That is just in the worst case of optimizers that resist every hint that their behaviour is taking all the fun out if the game.
But increasing the difficulty for everyone is just making things worse for the non optimizers.
At some point you need to let the optimizers know that the enemies (who in the story should have heard of the heroes) will send someone to deal with them, while the enemy's support cast deals with the rest of the party.
I am saying this isn't a good solution. You
do not need to do that.
This assumes your
only levers are changing what NPCs and monsters do in the world. And that is not your only options.
Your position seems to be that the rules of D&D, the monsters in and NPCs in the world, and the treasure in the world are all fixed and out of your control as a DM.
And the only thing you control is the behavior of the NPCs.
This is explicitly not the case. You, as the DM, are not just the player of the NPCs and monsters. You are in charge of the world and how it works.
If you are playing D&D as if you are "the player of the monsters", then yes, this becomes your only choice. I am saying that this is not a good plan.
This is basically a trap.
Again, just one of the later options when difderent things fail.
And on top of that, that might be fun for the optimizers, making them think about their defenses and so on.
And fun for the rest who can deal with foes they can take.
Look at buffy: she is clearly the optimized character. The rest (at least for the first few seasons) are sidekicks with social or knowledge focus. More often than not they are decoys to allow buffy to deal with the real foe.
So there is an RPG bulit around the buffy universe.
In it, the slayer character has a bunch of mechanical things that nobody else can match. The slayer is awesome at slaying.
The support characters both get narrative abilities that let them impact the game, and have the ability to invest in abilities that the slayer doesn't have an advantage in (like knowledge, social, whatever).
But here, the game (a) seeks buy in, where only one player plays the slayer, and everyone knows they are the combat-god, and (b) has mechanical systems to make it so that the other characters get spotlight time as well.