Yeah,
My group said they loved puzzles and so I added some logic puzzles to a wizard's lair and it was the least fun they had the entire time we played. Something about it being "on the clock" made the puzzles more stress than fun. I stuck with exploration puzzles (like finding hidden switches...