It's not really that. It is merely that for choices to matter there must be some constraints and limiting the players ability to affect the setting to that of their character is one possible constraint. People seem to recognise this in a sense that they accept that the game needs to have rules; that a situation where the players can just freely declare anything and have it be so is not optimal. I really want people to answer this: do rules that place limits on how the player can affect the fiction or 'gamestates' reduce the players agency? And if they do, why we have such rules?
I'm going to try to answer your question here but go a bit further. Rules do a lot of things:
CONVEY PLAY PREMISE/PRIORITIES
Are we testing players' skill in extracting treasure from ruins while that crucible reveals/evolves the nature of their PCs or are we finding out how what gun-toting Paladins will do to uphold the Faith and mete out justice in a fantasy Wild West that never was?
CONVEY GENRE
Laser sword wielding ascetics with supernatural powers and swashbuckling space opera or dark fantasy apocalypse Peaky Blinders.
STRUCTURE PLAY
TTRPGs aren't free-form so we need to know how conversation is supposed to unfold, what props (if any) and when/how to deploy them, when to consult the dice (or whatever) to find out how the gamestate changes, when to write something down/tick a box etc, how do reward cycles and attrition work and advancement/PC change resolve?
DELINEATE PARTICIPANT ROLES, GIVE AUTHORITY, AND TAKE IT AWAY
Why we need different people doing different stuff (playing obstacles/adversity and playing protagonism), how much latitude does each participant have to make a thing happen, when and how does that change during the course of play, what is "the system's say" when the agenda of two (or more) participants collide?
I'm going to start with your first statement but revise it a hair:
For choices to matter there must be some constraints/limitations on all participants.
If the apex play priority of a game is about testing player's skill at x, then a referee who has no constraints on their authority will create 1 of 4 possible persistent states at the table (or any 2, 3, or 4 simultaneously with enough players):
* Player's choices ACTUALLY don't matter because the GM will use/has used their unbridled authority to manipulate outcomes at their discretion.
* Players exist in a persistent state of insecurity because their choices may (or may not) matter in any given moment, but they can't be sure because the GM has mandate to leverage the offscreen/backstory (that only they are privy to) or ignore/change action resolution results to manipulate outcomes at their discretion.
* Player choices matter because the the players have extended trust to the GM to respect outcomes despite their mandate (whether the GM has authentically earned it by actually respecting outcomes or contrived it by being highly proficient at Illusionism and/or their players aren't perceptive/invested enough to detect it).
* Player choices are irrelevant because the players just want to feel like their choices matter and their skill is tested...so long as the GM is capable of manufacturing that state of being then the player is happy enough to go along with whatever is happening.
Now I'm going to go back to the Moldvay example that I wrote out above.
Moldvay's (pretty much) exclusive play priority is testing tactical (Turn decision-point management in exploration, Round decision-point management in combat, creating and managing class synergies, etc) and strategic (loadout management, long term resource - rest/recharge etc - management, when to parley and when to fight, when to egress from the delve and when to push on, etc) skill.
If that is the apex play priority, then authority by any participant (GM or player), within any of the outlined components of rules that I've outlined above, that disrupts the competitive integrity of play with respect to that priority DOES NOT increase agency. It reduces it.
Put another way, if some facet of system/rules isn't distilling skill from ineptness, but rather distorting it and/or making it impossible for the cream to authentically rise to the top, then that facet of system/rules is rendering play fundamentally incoherent. Agency is decreased because agency is "play-priority context-dependent."