Ogrork the Mighty
Explorer
... the less they seem like impenetrable fortresses in a world with dragons, gryphon-riders, and spells.
And therein lies the flaw in the original question.
Real-world Castles, more often than not, were not meant to be "impregnable fortresses." It would be great if things turned out that way but the use of castles was to serve as strong points from which local forces could sally forth and engage the enemy (who would not have such protection to retreat to).
Castles were meant to secure territory but if they didn't have a capable garrison then the enemy could just plunder and raze the surrounding countryside and keep the garrison bottled up inside the castle. Still, the garrison would be a thorn in the attacker's side that could not be ignored (consider what would happen if reinforcements arrived; they'd have a secure base from which to operate... not a good thing for the attacker).
The idea that castles should be "dragon proof" is, IMO, a flawed one. Castles weren't designed for that so it's no surprise that they aren't as effective against dragons. That being said, it's better to be in a castle than in an unprotected village.
Just because there are fantastical beasts doesn't mean there won't be "normal" threats that the castle would be useful against.
In my mind a dragon is just a different form of siege gun that, if you let it, can tear a castle down piece by piece. It's all in how effective the garrison is in combating the dragon. Specifically, the garrison should be taking the battle to the dragon ASAP. And a dragon must sleep sometime...