People who are discussing a game like D&D view player agency within that game in a certain way. To them, the idea that a player has authority over the fiction is anathema, just as the idea that the GM would have authority over the player's declarations is anathema. On the other hand, there are games in which it is perfectly acceptable (and encouraged!) for a player to have authority over the fiction- and there are also games in which it is perfectly acceptable for the GM (or even other players) to have some authority over the player's declarations and history.
Yes, I certainly think that any determination of "agency" (which is a pretty nebulous term) is entirely dependent on the framing of the scenario of agency.
Like, I would say a starting pitcher has more "agency" in baseball than the other position players, because the starting pitcher has by far the most control over the course of the game, as they touch the ball for every pitch on their side.
But, I would also say that a basketball player has more overall agency within their games than a baseball player, as a basketball player can impact every possession within the game, both on offense and defense, and the fact that there only 5 players per side in basketball compared to baseball's 9.
But (again!), I would say that a baseball player and a basketball player both have equal agency
as people because they can both choose to ignore the rules, disrupt the game, or just leave entirely. When you leave the frame of "competitive sports", then the difference in rules between baseball and basketball cease to be relevant.
For a simulationist type game, one player assumes more agency (the DM), and the other players assume less
precisely because the demands of the play style require it. Simulationism play is based on a deliberate eschewal of agency over the fiction by the non-DM players in the pursuit of a specific relationship between those players and the fiction. The players have less overall agency in the frame of "all the various types of TTRPG play", but they have the exact amount they need in the frame of "playing a simulationist TTRPG".
Just because a position player in baseball has less overall "agency" than the starting pitcher doesn't mean baseball is broken, or that baseball is somehow worse than basketball.
(I mean, basketball IS better, but for a host of other reasons.

)