Without using fancy mathematics, I made a confusing dungeon once.
It was a little demiplane. It consisted of rooms, with several doors, all floating in limboplasmic stuff. Well, the "doors" were in fact transportation devices. Each time you opened one, it created a corridor with a random length, and another door at its end. This door at the end would connect to a random room.
It confused the players, until one figured it out. Then they stayed in a corridor, and opened/closed the door until they arrived in the place they were looking for.
Fortunately, the previous levels of the dungeon were more conventionals. Maze, caverns, insane labs, etc.