Depending on the party size it'll take him about 40 minutes to cast it, ample time for a monster or two to come on in and attack a weakened party. As a DM if this was constantly abused, I would also have a bunch of monsters set up shop right outside the wall, not knowing that there were adventurers there, but neither would the adventurers know the monsters were there. Alternatively...
"Hey Bob, you wanna know something? I've always hated this stone wall."
"Yeah, why do you say that Bill?"
"I dunno, I just think the dungeon would flow a lot better into the torture chamber if we tore this wall down."
"Ya know Bill, I think you're right. Let's tear it down right now."