It doesn't sound like this ruined your game. Were the players happy with how the game went? If so...then it helped your game.
Pretty much this. I mean, yes, definitely have people try to hit the person who cast banishment, since it's concentration duration and thus can be disrupted, but... your players used a spell cleverly and did well. Sounds good!