In my games the exact nature of the wall's climbability doesn't tend to come up unless the player shows an interest in climbing the wall. It would get rather old having to narrate the climbability of every wall I ever introduce into the game. I suspect your game functions in much that same way. You primarily elaborate on details the players express an interest in. In which case the wall's details only get narrated when the player express an interest in interacting with it. It's simply a method of framing.
What I personally would object to is not if the DM framed a wall as a solid sheet of ice after I expressed an interest in interacting with it, but if he framed every wall ever encountered that I expressed an interest in climbing as unclimbable. The issue there for me isn't agency, but immersion. Doing such would instantly break my immersion. *Now if we were in an ice filled cavern then I'd again be fine with every wall encountered being unclimbable, at least without special gear to do so. Why the difference? Because the issue isn't about agency for me, it's about immersion.