For me, the particular frustration comes from the fact that the only actual increase in difficulty is time spent at the table before getting to the good part.
I mean, in the real world, if I have to go and get lumber from the other side of the village and carry it back to the building I'm trying to break into, that makes things more difficult.
But in the context of RPG play, that is just more stuff that I have to play through, taking up time at the table, with the likelihood of anything exciting coming out of it being rather small. It's busywork, the non-combat equivalent of filler encounters.