So, thanks for the replies all! This kind of blew up while I was sleeping, so there's lots to respond to here. Others have already answered queries about overall difficulty (insanely high, which was
not the source of player complaints) and observed that the players had the option of freely walking away [1]. It's fair to note that the triple high level spellcaster is a pretty hellish thing to face, but I think we should also note that it wasn't the concern expressed by the players.
The party is actually winning this fight, I should note. We had to end with it still ongoing - 3 hours of dice rolling wasn't enough, sigh - but the players have killed both dragons, both Kr'y'izoths (16th level Liches) and one of the Tl'a'ikiths, with the other on 12 hp, while Vlaakith herself has no spells above 5th level left and is down to 240hp. Even with the party heavily battered (so that they've each lost ~100hp, having used most resources) and down two party members, I think it's fair to say that they'll win in a couple turns unless the dice go abnormally badly for them. My tactics got very shaky towards the end of the fight - it was a mentally exhausting one to run - but with Concentration used on the Monk's maze it is unlikely that I'll be able to do anything particularly clever beyond just spamming high damage cantrips and counterspelling anything the Cleric tries to do. The party got to this point by using twin anti-magic fields and splitting up enough that AoEs were worthless, while focusing down the threats one at a time.
The Power Word Stun was abnormally effective because the Rogue has Con 8, and it's a DC 21 save. So, yeah. The Paladin eventually ended it with some special ability of his, but he had to kill a Dragon to clear a path. The Cleric didn't help since he was inside his own Antimagic field. Them splitting up so much, and using two separate Antimagic Fields, had the result that I had to rely on single-target spells to affect the three viable targets, meaning that I was using save-or-suck more than I normally would. Hence why they got hit by such potent ones with such frequency. I actually picked Prismatic Spray just because two players were standing in a line, and it was the highest level line or cone effect that I could find; I didn't even know it did banishment effects!
Uhm, I've talked too much again. But in short; the players were complaining about being hit by these spells that they often had no realistic ability to escape from quickly. We might fairly say that their complaints were touching on the way 5e saving throws are allocated. I countered by observing that these are player spells, and that I'm loath to nerf them since that starts us on a route of nerfing all player abilities; but the player in question responded that what was fair to use on an NPC is not necessarily fair to use on a PC, because it might mean a player gets to do nothing. This is... true, but the consequence is that NPCs never use very common PC tactics like stun-lock or banishment.
[1] I often have BBEGs make a boastful, "Why don't you flee now?!" comments, so the players usually ignore it. However, they were actually fairly tempted by it this time. When they walked into the room, and the illusion covering the bad guys faded so that everyone could size each other up, Vlaakith stood up and said, "I know you are here because you want to free the Hobgoblin slaves. If you walk out of here now, you can achieve that goal - I will free them to you directly - and nobody has to get hurt." They trusted her word, and gave it due consideration - but they decided that they wanted to kill her regardless. It was an interesting moment though, watching the players decide whether they wanted to just skip the rest of the dungeon I'd spent days preparing
or possibly "Metal."
