Actually, the reason I don't worry about the collateral damage of spells is that they have very little. The rules for material hardness and hp make it pretty much impossible to destroy stuff without a concerted effort.
For instance, Fireball, used outdoors, makes a crater only 1 inch deep...