5th level casters in the Forgotten Realms? Probably thousands. Which does mean just what you said further down...How many high level casters are there?
I personally do assume magical defenses that are not necessarily in the book. This was more of an issue in previous editions when a rock to mud spell could take out a castle from a distance, but just because spells aren't explicitly spelled out in the PHB does not mean that it does not exist. Spells in the book are just spells appropriate to adventurers, spells that for the most part can be cast in a matter of seconds.
A world like the Realms would have all the magical military defenses against Fireballs because as soon as someone invented the Fireball spell... some other wizard would have been paid by a nation to develop a counter. Which was basically my response to @Reynard 's question as to whether soldiers would line up in formations and such, or spread out to avoid AoE spells. In a world where there would be enough mages of 5th level to line up for a nation's military conflict (or even more... knowledge enough of the magic of the Fireball spell for nations to spend all the time and money they need to create Fireball wands up the wazoo to allow for long lines of magical artillery)... the idea of medieval combat would have gone out the window. Armies would be run and fought a la First World War, not the Dark Ages like most D&D players visualize it.
Most giants could do a significant amount of damage to castle walls. Just like all sorts of siege equipment from catapults to trebuchets could. Except those siege engines are probably more difficult to "kill" and have greater ranges.
But trebuchets also do not walk over castle walls, and do not cause damage at the same speed at which giants would. Loading up a trebuchet takes time... a half-dozen stone giants rushing a small city does not. A city would not spend their time and money building plain stone walls for defense when they can be much more easily breached... especially with armies that can field flights of military atop griffons, hippogriffs, wyverns and the like.
There's a reason why cities nowadays do not build walls around themselves... because the ways to get over said walls to attack the city within are so simple to do that walls are pointless and become obsolete (thank you Wright Brothers!). And with the amount of magic and monsters in typical D&D that could walk, destroy or fly over those city walls... medieval city planning and construction should also have been rendered obsolete.
The fact that it isn't that way is purely because we players love the idea of castles, knight, and fantasy. But based upon the "technology level" that magic spells can produce in said typical D&D worlds... medieval construction would have been blown past centuries if not millenia ago.
Just wanted to pee in everyone's punch bowl a bit.
