I think it would be super-fun if high level spells took multiple/many rounds to cast, and the martials had to protect them until complete.
It is not fun for the spellcaster if it is all they could do.
I think banishment has it right. Minor effect for 1 minute, concentration.
If you can concentrate until the end, boom.
Does not have to be 1 minute. 3 or 5 rounds as a doom clock would suffice.
Not having a different concentration spell up would probably enough of a cost.
If it is not, you could reduce the caster's movement speed to 0 too. Maybe limit them to only cast cantrips while they concentrate.