As a DM and regular spell caster, I wouldn't make a regular habit of this because your spell caster may quit the game. I think every once in a while would be ok. Maybe the spell caster's focus is stolen and disappears. Finding it opens up a whole new quest for the team. Maybe during the quest...