Varying forms of concealment would help, such as displacement, obscuring mist, or even entropic shield (I think). Bumping up your touch AC to make the caster miss is also a good choice. Alternatively, you could try to keep him from casting, such as using counterspell, or mastery of counterspelling, improved counterspell, and/or duelward.
But, quite honestly, you've already let the cat out of the bag by allowing the orb spells as is.