It does work this way, and you both need to look at that table again it isn't the party level. It is the level of the challenge. See pg 503 the last paragraph in column one (last sentence).
"Or you might decide that the 15th-level villain who created the dungeon crafted the wall, and use the...