The effort to turn RPG guidelines into hard and immutable rules is as old as the hobby itself.
I think the OP's excellent argument that you cannot celebrate player agency at the same time as structuring your gameplay to remove said agency solid. Note that you can, in fact, do all the points there in something like D&D 5e. You can have players create characters with concrete goals, and group ones. You can be open about resolving tasks towards a larger conflict that is in service to the player's goals. You can take your cues of where play goes from the pre-stated pass/fail events. You can ask questions about the world, allow players to establish fiction or use knowledge/perception rolls or whatever to help that happen. The design might fight you some, but it's totally doable (I've done it and had players say stuff like "wow this is what it looks like when a DM actually means Im giving you agency?").
The difference is that you need to opt into it, and have players who are on board with working outside the baseline system a bit; since 5e doesn't give you the same hooks to grab or conflict resolution mechanics that are much more clearly player-goal oriented. 4e did so, and games like Lancer/ICON etc show another way you can meld really tight combat with strong player-agency centered narratives.
And I also concur with OP that lots of people dont really want this play. Many are quite happy with enjoying a story mostly told to them, with a little bit of their backstory laid in & some space to do 3 hour shopping expeditions or tavern scenes to reclaim some agency for a bit before relinquishing it again.