That said, this would be a partial nerfing of casters and their speciality, so it shouldn't be used often.
Good idea, and I agree strongly with your last sentence. I might do that once (or twice) during the course of a campaign unless circumstances called for it to be used more often. Otherwise, the PC playing the caster might feel you're targeting his or her character.