Precisely. Keeping mechanical representation rigidly fixed, and simply hoping that changes in player stats will by happenstance result in the monster transitioning from "terrifying threat" to "minor annoyance"....just doesn't really get the job done. It's not that there isn't a theoretical way for it to happen, I'm sure someone could come up with some (dare I say it) white room arguments to the contrary. But in practice, it's just...not how 5e ends up playing.
I suspect this is, at least in part, because HP has to bear effectively the whole burden of representing monster scaling, given AC is damn-near static once you get to ~20, but damage output scales in a pretty chunky, uneven fashion and much, much more slowly than HP. Creatures at the absolute bottom end of the progression (e.g. CR 1 or less) may be usable aplenty, but once you get to CR 4 or so, things plateau so much that it's just too much work to use lots of those monsters. Typical HP for a CR 4 creature is well over 100 (reportedly in the 116-130 range), and even a well-equipped high-level Fighter simply can't cleave through that in one attack.
Even if we assume a Fighter has a +3 weapon, +11 attack bonus, 4 attacks per attack, and hits with every one of those four attacks (which will be much higher than actual expected damage output, even accounting for crits), that's at best 8d6+32 = 60. It takes pulling out all the stops to potentially take out a CR4 creature in a single hit as a level 20 Fighter, which everyone always tells me is the king of damage and should be absolutely destroying everything. Heck, even throwing in GWM without any accuracy penalty (which would definitely be an exaggeration of damage), that's still only 100 total damage--a Fighter who lands every single GWM attack cannot consistently kill a CR 4 creature in one round.
So we're stuck having to work with pretty much CR1 and CR2 creatures exclusively if we want anything like "minions," and that roundly interferes with the "this enemy, who was once a deadly danger, is now barely chump change." Further, typical damage is in the 27-32 range, meaning ~29.5 damage on a hit. Even if we assume a low but moderate hit rate (typical attack bonus is +5, player AC is rarely higher than about 18 without a shield, so 40% is on the low end of reasonable), a swarm of, say, six of these creatures is going to get around 71 damage in on the first round, and probably another 30-40 on the second....when a 20th-level Fighter, we'll assume maxed Con, only has about 15+11x19 = 224 HP. So, despite this being only a quite "small" group of "weak" creatures, going up against one of the allegedly strongest characters around...you're still looking at taking a major beating (albeit possibly distributed across the party) and not actually cleaving through these things quickly at all.