"As long as the wall holds" is the flavor you're after?
Well, for scrying, you can just lead everything. A thin sheet of lead stops most divination effects. But False Vision (Sor/Wis 5) is more fun. Especially when combined with Mirage Arcana (also Sor/Wiz 5) somewhere else to attract the Scry/Buff/Teleport crowd. Surely there's a nearby volcano that can be illusoned up to look just like the fake vision that's plastered of the inside of the castle to anyone who checks....
For Teleporation, the lowest Core counter is Forbiddance - which is somewhat pricy. Keeps out Teleporters, though.
If you lead the walls, most Scrying spells fail.
Houseruled material: Perhaps there's a particular stone with some antimagic properties; it's expensive, and so is only really tiled an inch thick on the inside of the building where it's safe from direct assault. The way it works, the stone itself is imbued with an antimagic field (that only applies inside the stone - a Wizard could pick it up and cast a fireball, no problem, but the fireball wouldn't burn the stone). An incorporeal undead, or someone using a spell, spell-like, or Supernatural ability to become something that can go through stone will have severe issues (hand/foot/head turns back to normal inside solid rock) and a single disintigrate won't breach the wall (although it will expose the antimagic tile layer to more mundane attacks, which can breach the antimagic tile).
Remember the Line of Effect rules - unless there's a full square foot of breach in a given 5x5 square, you don't get line of effect. An arrow slit two inches wide and five feet tall is only a 120 square inch breach - and it takes 144 to pass that Charm Person or Lightning Bolt. You can shoot an arrow out through it no problem, though. If it's no more than 2 inches wide, it can be of any height (as there won't be more than 120 square inches open in any given five-foot segment). Fireball and it's inheritors get a partial workaround (it's got a small hole clause), as do certain effects (Demand - which requires a piece of the subject), but almost nothing else does.
You'll also need a cieling (you don't want people flying in) or some way to fake it (towers with See Invisibility and Glitterdust).
Add a little something for the Reduce Item boulder trick, and you're fairly good to go; in order to get through, they have to get through the wall (which can be physically destroyed - either by direct damage such as from seige engines, or by a Disintigrate followed by much less direct damage, or by undermining the walls - you've got defenders actively trying to stop you in the meanwhile, though; a classic siege situation).