D&D isn't going to perfectly simulate the world for NPCs. It's rules strictly for adventurers, and that's it. You don't need to deal with all this stuff in the rules. That is how you get too many rules for what's supposed to be a relatively easy-to-learn role playing game.
Fine. Substitute "1st-level wizard" for "1st-level commoner." The problem is the same. It's not just adamantine doors--the wizard can dig out of, or into, a prison cell; can dig through a castle or a dungeon wall; you name it.