A lot of the things that are often brought up as good aspects of dungeon maps are things that I think are great about 90% of the time, whatever the context (though size/sense sometimes restricts the inclusion of all), provided I'm running something bigger than a simple lair:
- Multiple entrances/exits
- Nonlinear/looping pathways/"Jaquaysing"
- Use of verticality and varied slopes
- Disruptive natural elements (subterranean rivers, crevasses, etc.) cutting through whatever grid is there
- Places you can see but can't obviously reach without further exploration
- Mixture of small and large spaces, chokepoints and open areas
- This is verging on dungeon stocking advice, but I want a mix of relatively empty and "busy" areas
- One way gates and portals that interrupt normal spatial navigation
- Structure that can be reasoned about ("symmetry implies a path/secret door here", or "this ecosystem/structure likely has a water source")
- But enough asymmetry to make things interesting
In general, I want to promote interesting tactics both by the party and the inhabitants (flanking, chokepoints, ambushes); encourage exploration, navigation and longterm thinking beyond moving from room to room ("we see that area across the crevasse, how to we get there"); reward players mastering the space (e.g. being able to use the layout against enemies, or thinking about the "archaeology" of the space to find things, like "a mine probably has vent shafts"); and often leave open the possibility of getting lost (one way teleports are useful for this).
I have tried to use real world castles in play and they just don't work well for what D&D is trying to do. I don't think realism in dungeon design creates fun play.
I can imagine this for castles but there are a lot of great real world subterranean structures that I think would play quite well. This is part of the underground city of Naples (purple), I think it would make for a nice dungeon layout, maybe "gridified" for convenience:
(Side note: in case anyone tries to dispute the plausibility of large underground structures everywhere in your setting, just remember that Rome, Paris and Naples all basically sit atop multilevel megadungeons).
As far designing dungeon layouts that are both fun and make sense in the real world, I think this is usually a place where you can have your cake and eat it too. I've rarely found these two things in conflict (more often, they synergize). While most real world castle layouts might not provide the best gaming, I think you can usually throw together a modification on the theme that keeps most of the elements players would expect (and critically, can tactically reason about) in a castle, and add in just enough gaminess to make it fun without busting verisimilitude. That said, sometimes I find it healthy to throw out sense and just chalk weird stuff up to the nature of the underworld.
