The Banish opens the door for auxiliary encounters within the greater encounter. Here's what I mean:
- Dropping a PC into a ravine and forcing them to use a series of ability checks to get out. Works for lava and stuff too.
- Dropping PC into a pit filled with monsters, they have to fight their way out. Use minion rules to make it quick.
- Dropping the PC onto a collapsing building. Saving Throws to survive, ability checks to get out.
Etc etc. So I think if you are playing optimally, also think, are you playing in an engaging way? Optimal stuff from the DM is easy to do IMO. But optimal + stylish/engaging is much harder. You're a very experienced DM so I trust you already know this, but just wanted to build off your idea here.