If you look at the NPC mage in the MM, it's a CR 6 monster that has 9d8 hit points and casts level 5 spells. If you compare it to a standard level 6 wizard, it has a few less hit points and its spell save DC is slightly lower, but it makes up for that with greater invisibility and cone of cold. We don't have a full suite of NPCs for every role and CR, but if we did have a CR 6 archer and a CR 6 barbarian and a CR 6 druid to round out the NPC group, then I would expect it to fare about as well against a CR 6 dragon as a level 6 party would - which is to say, they would stomp all over it.
You want me to admit that an optimized group of PCs is effectively higher level, and should be fighting correspondingly tougher monsters, and there are two problems with that:
1) That's not how the game works. The game says that the strength of a party is measured by its level, and challenges should be measured relative to that level. Moreover, it would be terrible if you did make the monsters stronger to compensate for the competence of the party, because it would render player choice meaningless. If the high-Charisma barbarian with low Strength only has to fight one kobold, but the high-Strength barbarian with low Charisma has to fight four kobolds, then there's no point in playing a competent character.
2) Even if you don't care about invalidated player choice, there's nowhere left to escalate at high levels. Monsters can only get so tough. Ability scores only go up to 30. We know how strong the Tarrasque is supposed to be within the world, and we know the numbers associated with that, and anything that could threaten a high-level party (numerically) would melt the planet in half (narratively). Maybe if we got some stats for some actual gods, then we might have some context to provide a real challenge, but unless the PCs are the only high level adventurers in the entire setting...
now this is getting way way way out there.
the game does not say the strength of a party is measured by its level. it provides a rough mechanic for tabulating the expected results from a combat making some very broad assumptions.
if anyone, much less te game assumed "four level 7 characters" had some defined strength of say 7-thumps, then that would mean it would be equal to every other four level 7s which also have a strength of 7 thumps and that is just an insane position given the variety of choices in build and design that players have, not to mention the large amount of variety and variability in encounters. Lotsa undead vs party with no cleric = tougher than lotsa unded vs party with three clerics - just to name one.
your choice to try and read these things as somehow locked in stone absolutes for the purpose of your fabrications is tedious.
But, far as i know, a great many Gms in a great many games have tailored there games thresholds and challenges and so on to fit their party for decades upon decades and amazingly players have not rebelled for being rendered "de-choicified." Cant speka for everyone but most of the games a set of players enjoyed being challenged even after they "got gud" or whatever.
As for the nebulous "whats the incentive..." the incentive for all things and choices in this game is because you enjoy it.
one player optimizes the hell out of his character, seems outputs in combat that are great... happy.
Another player doesn't and scores less damage in combat but is happy because the choices they made were focused on other things like their style or roleplaying focus... happy.
Maybe you have experienced lots of players who wanted to optimize their characters and then expected to see easy encounters cuz they out-designed the statblocks in the book... but i have not and i don't know of most any Gm i have seen who advocated this as a thing... well possibly until now.
We keep coming back to - with the constraints you are chosing to wrap around the elements in your argument, you are creating the illusion of conflict within the game, but it is really a conflict of your constraints.
So, like the humorous doctor says when someone says "it hurts when i do this"... "well, stop doing that."
enjoy.your self-inflicted and self-intended crisis.