This won’t change much, it’ll just shift the player meta to either a) going for pure damage and nothing else, since spells aren’t worth the investment, or b) speccing the group into being really spellcaster heavy, so they can apply all three levels in one turn and then go back to the old playstyle.
That's selling the idea far too short.
a) The spells should still feel worth the investment, since even if a single Banishment or Dominate Monster doesn't equate to pressing the insta-win button, they should still inconvenience the monster.
Put more succtinctly: If the players feel spells aren't worth the investment, that means the implementation was done poorly. It doesn't mean the idea was bad to begin with.
b) you say 1) that like it was bad or something? and 2) you very strangely say "then go back to their old playstyle" as if the wizards could suddenly transform back to kick-in-the-door fighters?
If the group concludes they need three spellcasters (so that they can get in three Banishment or whatever in quick succession) what's wrong with that? I would have thought, however, a more balanced approach would be to shield the wizard for three rounds. Which would accomplish everything we want out of a Solo fight.
And I'm not sure what you mean by "the old playstyle". Once they have defeated the Solo, there's nothing to go back to. The fight is won. Please, feel free to explain.
PS. Please don't read my suggested solution of the three strikes and youre out as a fully fleshed out complete rules proposal.
For instance, I am well aware a single spellcaster might not be capable of pushing out so many Imprisonments or Dominate Monsters that the math tells us are needed if you need three to "lock down" the monster.
So it's eminently possible the framework should encourage DMs to interpret many spells as stacking together.
For instance, my Fear example earlier, with Shaken --> Frightened --> Panicked.
This isn't going to be a very high-level example, but still: Let's say you successfully cast one Fear, one Phantasmal Killer and one Nightmare spell. They could all contribute to locking down the monster, with the final third spell determining the exact results.
The idea is that all of these spells are in roughly the same neighborhood, both thematically and power-wise.
The idea isn't to require you to memorize three (or six, assuming a 50% save rate) identical spells just to defeat the big bad.
The idea is to allow save or suck spells to have SOME impact, without giving them ALL the impact.
As a player you should feel casting save or suck spells is worthwhile. There must be some middle-ground here, and not "unless my spell insta-wins the combat I'm going to resort to damage only".
Especially if, as my plan is, you make sure Solo monsters have lots of hit points.
What we all want out of a Solo fight is a monster that might be killed off by hit points damage, but where spells are needed to make the monster suck just enough to not kill off the fighters.
Or, conversely, the fighters managing to keep the monster from killing off the spellcasters (or themselves) long enough for the spells to grind the monster down. Which should not be about a single roll of the die, but something that generally takes a while.
What we all want is the math of the game to usually lead to a nice satifying climactic fight.
Nothing wrong with the odd one spell and you're out fight, or the we-all-critted-so-the-monster-didn't-even-act-once.
But the game's math currently only achieve an average fight of more than a single round by heavy handwavium, and that's not a satisfying design.
