As a player, there is the Mage Slayer feat for martials. But generally, its really a fight fire with fire case. Silence, Counterspell, Dispel Magic, Globe of Invulnerability are your best friends. Also, having high saves would help a lot.
As a DM, just make some rocks fall on the caster...