It's an interesting set of observations. I'm assuming that area defense is considered so much more problematic because of the idea of equating magic with technological and scientific power/physics.
I don't think so. D&D's palette of traditional spells are focused on dungeon exploration and designed with balance with regard to dungeon exploration in mind. They were balanced according to percieved utility in those dungeon environments. The larger effect on society doesn't seem to have been a strong consideration, nor does there seem to have been any underlying theory on the complexity of the magic or the energy involved.
I think that area defensive magic was overlooked primarily for two reasons:
1) It would have seemed to have been primarily 'screw the PCs' magic which took away options from PC spellcasters.
2) The PC's themselves had no use for it in a dungeon environment.
I think that because in part, area defensive magic does exist but only in forms which are useful to dungeon exploration and general adventuring. If you look at what we have in area defensive magic, we've got alot of stuff, among other things: Alarm, Arcane Lock, Antipathy, Consecrate, Dimensional Lock, Fire Trap, Forbiddance, Guards and Wards, Hallow, Ironwood, Misdirection, Faithful Hound, Magnificent Mansion, Private Sanctum, Nondetection, Obscure Object, Sequester, Tiny Hut, Zone of Silence, Zone of Truth. But pretty much all of that is designed toward creating a temporary defensive 'camp' for adventurers on the move, or else for assisting in the defense of a place by a few skilled individuals. As a means of creating a more permenent defense it all fails various tests of one sort or another: duration, availability, area of effect, cost, and/or livability. An arcane lock is really great for a wizard defending his tower, but not nearly as useful as a spell lock which can be opened using a more general key (spoken word, wearing a particular uniform, time of day, etc.). Faithful hound is an amazing spell for defending a doorway or passage against invisible invaders, but its duration is geared toward 'while the mage sleeps' and its availability (5th level) makes it problimatic as a common defense against the much more readily available 2nd level spell (Invisibility). And so on and so forth.
It seems to me that for game purposes, its logical to assume that a particular magical attack can be countered by a defensive spell of similar level. Likewise, we have a very obvious reason for wanting to imagine that the fantasy world has alot in common with the real world and that the impact of magic need not be so great on the setting that it becomes unrecognizable except by design. It seems to me that balanced defensive spells can be created that fulfill these goals.
It's important to note that I came to these conclusions not by being a DM, because the DM can create whatever defences he thinks he needs by fiat. Cost and practicality are no real obstacles to a DM because the DM has effectively unlimited resources to create suitable challenges with. I came to these conclusions being a PC, because as a PC you soon find that adequately or even minimally defending your possessions and retainers from magic with available techniques is cost prohibitive.
It's not just castles. Castles can be made to defend against dragons and fireballs without too much fuss, but they are difficult to impossible to secure against invisible attackers and dimensional travel. Sailing vessels by being mobile do better there, but are almost impossible to secure against magical fire attacks. And so forth. It's not just a matter of sufficiently high level spell casters can wreck havoc. That doesn't bother me. It's that a spell-caster of a given level has trouble frustrating attacks by a spell-caster of lower level because there aren't suitable defensive long duration spells available to do so. They need not be as proactive as Faithful Hound. An until discharged duration spell that cast glitterdust on invisible interlopers would do the trick. A spell that cheaply granted long duration fire resistance to objects according to their weight would go along way toward suggesting how societies delt with pervasive ability to conjure fire.
Then again with sufficient magic, or with cooperative magic, need castles be fixed emplacements as we normally think of them? Could they not become mobile command centers, perhaps even a type of FOB able to be transported or teleported into and out of the areas of an engagement as necessary? One could even imagine, with enough power expenditure, or trasnformative magic, a walking castle, or one that transforms into other things.
I don't have a problem with such legendary scale things as a flying castles and buildings that walk about on bird legs, I just have no desire for them to be so prevasive that they first lose their sense of awe and wonder and secondly transform society beyond the point were it can be easily related to by the player or easily imagined by the DM.
