Beyond grappling to pin and grab away the spell component bag, and bull rushing against a wall, I recomend you to always carry tanglefoot bags or nets, and use them when next to the caster if possible. Almost surely the caster will become entangled, which, among other things, reduces his speed...