If you build a fortress based on magic, then one successful "dispel magic" or "antimagic shell" and the whole thing could come down like a pack of cards.
I think "ùìéè äîáåê" made a good point about the need to be mobile. Magic could be used to provide near instant physical defenses, so there...