Set, I like your thinking and that is exactly the point I was trying to make and the innovations I was trying to elicit. My point is not that castles are useless or set decorations for a game, but that in a fantasy world they would have to adapt and look somewhat different to the way we traditionally expect them to look.
Some posters have argued that a fantasy castle need not look any different than a medieval castle for the reason that, even in a fantasy world, fantastical threats will be uncommon and the majority of threats posed will be traditional threats faced by medieval societies, barbarian raiders, goblin hordes, etc. which are not that different from human foot soldiers. While I think traditional threats are indeed some of the perils, perhaps even the most common ones, that fantasy castles must guard against, I think it misguided to hand-wave away the prospect of more fantastical opponents as "uncommon."
History has shown us that whenever a superior military technology or tactic has been discovered, it is quickly adopted and becomes not only common but the norm. There is every reason to assume that if there are creatures that can be easily bred or recruited, trained, and used for military purposes that they will be used for exactly that.
Medieval armies had cavalry; even very ancient armies used chariots and horses. Raising a herd of pegasi or other flying steeds can't involve all that much more cost and effort, and if the creatures exist and can be used to advantage, armies will do so. It is almost a given that flying mounts (or flying soldiers) would be an integral component of fantasy armies. Even such terrible creatures as Dragons could be selectively bred, over time, for temperment and controllability, so as to domesticate them--at least for the purpose of military conquest.
Any dark lord worth his salt would make sure that his invading army is as nasty and ferrocious as he can muster. To the extent that he can employ uncommon tactics, tactics which the defenders are not anticipating, and which their castles were not built to guard against, he will do so. Smart evil overlords will use every tactic at their disposal, no matter how sneaky or unconventional. If it is possible in that world, you have to assume it will be in their arsenal. I would expect even the lowly, local goblin horde to have bat-riders or to ally with cloakers to a attack a castle from the air.
To that end, I think castle builders will need to anticipate not only conventional attacks, but as many unconventional attacks as their imagination can devise. With that in mind, I can conceive of a number of potential opponents that fantasy castles would have to address. (Not including spells and magical effects, as I think they deserve a separate post):
Flyers: Pegasi, gryphons, hippogriffs, manticores, chimeras, dragons, drakes, rocs, giant eagles, giant bats, giant wasps, stirges, harpys, fiends, sylphs, gargoyles, cloakers, sprites, angels, air elementals, etc.
Burrowers: Dwarves, gnomes, bullettes, ankhegs, giant moles, dragons, and other diggers could be used to excavate tunnels underneath fortifications allowing access not only for them but for the normal legions that could swarm in behind them.
Phasers, and Incorporeal creatures: Earth elementals, xorn, ghosts, vampires, air elementals, etc. that can phase through rock, walk through walls, turn into gas or mist and seep through windows and cracks. All these methods frustrate the protection afforded by thick walls.
Diminuitive creatures: Sprites, sturges, insect swarms, snakes, creatures that can turn into insect swarms, shape shifters that can reduce their size and then grow big again. All these creatures, especially if they can fly and swarm in large numbers, can present harder targets to hit, are harder to spot, and can gain access through arrow slits, chimneys, drain pipes, and small openings that larger oponents could not squeeze through or that castle builders might not ordinarily consider as a vulnerable point of easy access.
Liquid creatures: water elementals, lava creatures, battle oozes.
Invisible creatures: [you can't see the list 'cause it's invisible]
Extradimensional movement: Ethereal, Astral travel, shadowwalk, feystep, gates, portals, summoning, etc.
The architecture of a fantasy castle would need to be adapted to defend against these dangers and more if it were to be an effective redoubt in a fantasy world.