hawkeyefan
Legend
It seems as though a table at which it's the GM's responsibility to bring at least most of a world suitable to play in might qualify as the participants agreeing on the world. This is the way the tables I've been at recently have done it and there's been no shortage of player contributions to the world or character-driven play.
I think this all depends on how the setting is approached.
In most of my recent games, as both a GM and player, the setting of the game pre-exists to a large degree. It's not one of the GM's own making. As such, the players have access to information beyond just what the GM shares with them. I think in such cases, it's much easier to not think of the world as "the GM's world". In these games, the GM likely still has a lot of input on how the game world takes shape, but the players also have a lot of input as well. Most of these games have involved a significant session zero type beginning where we collaborate on a lot of the elements of play. Blades in the Dark, Spire, Stonetop, Galaxies in Peril... all those games involve collaboration at the start of play, and continually throughout play.
I think that tends to not always be the case when the setting is one of the GM's own design. Often when that's the case, there's a default feeling that the world belongs to the GM. A lot of times, and I say this from experience, the setting is something the GM is creating entirely on their own... not during play, but on their own time in preparation for play. I did this for years. And while there's nothing wrong with that approach in and of itself, it doesn't easily lend itself to player directed play. The GM is doing the vast amount of work on their own!
There also tends to be a lack of knowledge of all the ins and outs of the world and its people and institutions and the like that may, in my experience, lead to players feeling like strangers in a world that their characters are supposed to inhabit. I think that lack of knowledge and how it's handled is the primary concern. If I can only know what the GM tells me, then how proactive can I be as a player? The GM either needs to offload a significant amount of information ahead of play, and I have to actually absorb it, or else there needs to be some other means of allowing me to not feel like an alien exploring an unknown land.
There are different ways these things can be addressed. I tend to agree with @pemerton though, that a good default approach is to think of the world not as the GM's but as the group's. It seems like a solid first step toward the kind of play that @Yora is looking for in the OP.