While it's probably not unbalancing (so feel free to allow it), it would not technically work by the rules. Enlarging, shrinking, summoning etc. spells do not do damage.
If you try to enlarge an object in a space too small for it, the spell will fail rather than breaking the enclosing material.
Similarly, the hinges on a door would cause a shrink spell to fail rather than becoming damaged/broken themselves. If you allowed this, you might have players next trying to cast shrink spells on their enemies' armor/helmets.