Here's another idea:
Teleporting is very draining on the caster. For one hour after the teleport, the caster cannot cast spells.
This is not swingy, the teleport is reliable, and useful. At the same time, it's not something wizard likes doing, and you certainly don't want to teleport into danger.