"Everyone is involved in establishing the game world" is the description you think is best? How is that not worldbuilding?
I think the criticism is for a specific type of worldbuilding, with the GM as the sole or heavily primary authority on the world details to be used in the game. I realize that my use of the term is far more broad. However, I think that broader definition is far more suitable overall, and I do think it aligns with a less RPG specific version of the term.
I think that many settings are put forth as neutral, that's true. But I don't think that must limit player agency. It depends on expectations. I share your preference of having characters that are heavily invested in what is happening in the game, and tied to the events that are going to come up. But some players don't want or need that.
So in this sense, the expectation of agency is the deciding factor. You expect more agency for the players, so if a game does not have that, then you will likely not be satisfied with it. I can understand that. I don't know if I would therefore draw the same conclusions about the game system or style of play that you draw, but I can understand your preference. If you think that worldbuilding tends to limit agency, then I can understand your concern. I don't think it is nearly as ubiquitous as you put forth, but I can understand the concern.
But if you did not expect much agency....if it was a casual game that you joined on a whim, where everyone was playing a premade module or what have you, and gave little thought beyond their character other than class and race....you'd likely be fine with whatever agency you had. (I realize that you would likely not join such a game, but let's consider this a general "you")
The expectation on the part of the player is aligned with what the game offers, in which case, nothing is being denied.
If I have two kids, and one wants 4 scoops of ice cream, and the other only wants 2, I am not denying the second kid anything when I only give him 2 scoops.
Sure, there are some differences that are striking. I don't know if they need to be as great as you may think. There is nothing that prevents the GM from introducing secret backstory. The book uses an example that continues throughout the rules and it's very clear that the GM is expected to do exactly that. Yes, the material the GM introduces is expected to connect to the themes and ideas established in the Exposition Session, but it still allows for it. But I don't think that this fact will wind up limiting player agency.
But what about my comment where I said this is exactly how I've been running D&D for many years? You clipped that off when you quoted me, but I'm genuinely curious for your take on that.
Why can't D&D be played with an initial session where everyone contributes characters with goals and then world details that fit nicely with those goals and the themes that seem to be generated as a result? 5E's game mechanics are not strongly designed with this in mind, but the Bonds, Flaws, and Traits can really contribute a lot in this way. And then there is no real limit to what you can do as a group independent of the mechanics.