I mean, I don't think it's MEANT to be punishing the players, but IMO it is, in that THEY want to play one way, and YOU the DM want to play a different way, and you are forcing your desired game style onto the others?
Like, if this is a common problem in your group you may need a different...