Sure, but the term is pretty clear in most usages outside of this one. At the risk of sounding like a broken record, look at board games. They can be wildly different in play, randomness, all kinds of factors, but we can meaningfully compare "agency" in the sense of "how much impact on victory do the decisions of players at the table have?"
There are games that have essentially none, games that have a lot, and a lot of games that use variability in such a way that players have only a little. The term has utility in that context because the frame is clear. We're mostly fighting with competing conceptions of what TTRPGs are here, such that we can't come to an agreement on what our frame is.
I agree with this. The trouble, of course, is with the second part of the statement. "Player agency" does not have a clear definition in TTRPGs, and is instead just used as a statement to browbeat others about one's own preferences.
There is the side-problem that most TTRPGs don't even have a clear "win/victory condition," which makes the use of the term ... difficult ... in comparison to boardgames.