Heh... I only wish I had the rules memorized to the point where I would know off the top of my head that climbing that particular building the DM put forth shouldn't involve a check and be an automatic success.True, but they also set player expectations about what their characters can and cannot reasonably accomplish.
A shared understanding of the rules is important to me when playing, it allows me to better engage with the story.
If I want to climb to the rooftops and there are no extenuating circumstances, the rules say I can climb at half speed no check required. If the DM asks me for a check* out of the blue or is inconsistent in how they arbitrate climbing, it's going to detract from my enjoyment of the game.
* Unless this exception was previously discussed and agreed to