If you aren't, then your entire line of argument goes away. Because what's not important is whether an enemy force can fly over the walls and land on in the large open bailey. What's important is whether or not the enemy force can fly over and land on the tops of towers. Because in any high middle ages motte and bailey style castle, the bailey itself is designed to become a killing zone if anyone breaches the gate. It doesn't really matter if you breached the gate by going through it or over it, if you end in the bailey you are behaving exactly how the architects that designed those fortifications expected you to behave. A small force commando style entering the bailey doesn't require changing the structure or design of the castle to any large degree. The bailey, whether inner or outer, is still designed to be a killing zone.
The problem with the open topped castles is that castles weren't designed to be defensible from the top down. So if you could land on the roof of the keep, you could put the king in check without dealing with the pawns, castles, and knights guarding him. Or if you could land on the roof of a tower, you could breach the defense in unexpected ways compared to a defense designed to resist attackers attacking from the bottom of the tower up or which assumed individual towers represented "hard points" in the defenses that could command the surrounding area.
Hoardings of any sort inadvertently also resist aerial commando raids of this sort, leaving the baileys as the only place to land and thus leaving the only place to land a prepared killing zone.
In general and across time, a couple score archers and crossbowmen in castles always had a better chance against dragon or manticore attack than the same force standing in a field. It's not such attackers that threaten the viability of traditional castles. The real problem with fantasy air forces is that they can bombard the castle with stones from an arbitrary height above which torsion missile weapons and most spells short of weather control can't reach. This is a bit of a problem, but you can handwave around it if you care too, by making fantasy air forces rare, by making them counter each other, and by noting that by the end of the middle ages the outcome of sieges were foregone conclusions because of trebuchet's and cannons anyway, so fantasy high altitude bombing doesn't really change anything. The point of a castle was to force delay on the attackers until defenders could be mustered and arrive, not to resist an attack indefinitely.
Anyway, rant off. Pet peeve of mine, I admit.