Responding directly to the OP,
IME something which is important at
all levels, not just high levels, is to provide monsters with environments suited to their strengths. I know it seems obvious, but bear with me: as an example, I've seen Water Weirds used in two different scenarios. One was when I was the DM, and it lurked in a fairly large pool; the other was when three of them were arrayed in a pool of shallow water for a straight fight. Despite both fights being against a group of (five and four, respectively) 5th-level PCs, the Weird in the pool was significantly deadlier (drowning one PC before the party fled), whereas the group of three was defeated fairly easily. Finding scenarios to minimize the PCs' strengths and preying on their weaknesses provides a more natural challenge than simply throwing creatures to overwhelm them through high CR. (Incidentally, drowning is my personal favourite hazard in 5e because PCs with average Con can be dropped to 0 HP in 1 to 2 rounds, and casters generally can't do anything about it once they've been dragged underwater).
Something else I've found is that single monsters usually die quickly; pairs of monsters more slowly; and pairs of monsters with reinforcements even more so. Reiterating what @
OB1 said, adding reinforcements is a good way to challenge the party in an otherwise routine encounter. A single Young Red Dragon would probably be killed in 1 round without many resources expended. A pair of Young Red Dragons, while formidable, is still fairly easy for 11th-level PCs. However, if a group of kobolds shows up and attacks the party wizard, even at +4 to hit against 16 AC she'll take a few hits (more with Pack Tactics). She'd probably have to use a Shield to avoid most of the damage and a Fireball to take them out, and if that happens three times in a fight, she's spent all of her 3rd level slots and most of her 1st; while still doing nothing to stop the dragons toasting the party with breath weapons. Or grappling someone to fly away and drop them in lava. Or simply ganging up on the front-line fighter and tearing him apart.
If you're looking for really nasty combos, combining Magma Mephits and an Iron Golem is somewhat ridiculous. They use Heat Metal on the golem and/or heavy-armour wearing PCs (2d8 fire damage per turn as a bonus action! No save!), then they cover the area in Fire Breaths, and the party takes heavy area damage while the golem heals for each point of fire damage it takes. Not to mention that when they die, they explode for yet more fire damage (and healing for the golem).
Combining Vine Blights (attack and restrain on a hit) and Shambling Mounds (if both attacks hit, target is restrained and slowly beaten to death as the Mound moves) is also fun. With the blights' Entangle slowing down the party, the Mound's lower movement speed is less important.
As has been previously mentioned, invisibility is HUGE. Water Weirds, Invisible Stalkers, and others can really put the hurt on, while being immune to pesky things like opportunity attacks (Relevant phrase being "You can make an opportunity attack when a hostile creature
that you can see moves out of your reach," emphasis mine) and retaliatory attacks.
EDIT: As far as actual play experience past 11th level (and large parties), I'm admittedly limited. Having reached 10th in a party of 3 I'll say that casters hold most of the "I win" buttons, and it didn't seem like that was going to change any time soon when we ended up finishing the campaign.