Collective world building through narrative results in a story that, if it goes for a while and is actually mentally examined which it probably won't be, at best results in something like 'Lost' or 'X-Files'. It will be disjointed, rambling, and often incoherent and contradictory. Chekov's Guns will be introduced and rust unfired and forgotten. Plot lines will be introduced with bangs and then never touched again. Things that ought to have huge meaning and implication for the setting and the characters will appear by 'rule of cool', or as deus ex machina, or techno-babble resolution, or as McGuffins - and then disappear, their implications and meaning never explored.
Please don't talk about "at best" for a style you dislike. Everything you are talking about does happen at average. And all the things appear at average in DM driven campaigns - indeed I find the insistence that Chekov's Guns
must be fired to be somewhat harmful to a believable world unless you're moving at breakneck speed; it implies that the entire world is a facade put up for the PCs.
As for things being solved by rule of cool or Deus Ex Machina, in my experience that's most common in 90s adventure paths where the NPCs do everything and the PCs are there to bear witness. And it's a problem more common with specific DM authority than one with collective world building.
That said you are right on the nail when you say that some of the best games are ones with built in short story arcs rather than attempting to be an epic series. Fiasco, for example, is a recipe to create a Cohen Brothers movie in approximately the time it would take to watch one. Dread a horror film in the time it would take to watch one.
Which is where we get to almost a clash of media. Most good shared authority settings are effectively miniseries while it normally takes longer to play through a single author campaign than it does to watch the whole of Game of Thrones.
The cost is simple. You go from the experience of being within a fantasy story - maybe a bad fantasy story but hopefully a good one - to the experience of being one of a team of screenwriters hammering out the draft of a screenplay around a table. It's not possible to be in the story you are creating. GM's are never immersed in the story the way a player is or can be. If you make everyone wear that hat, then every one is interacting with the story primarily at the meta level.
This depends
which system you are using and how it's happening. There's a reason I claim that modern RPGs started with
My Life With Master. It was the first game I'm aware of to have an inbuilt character arc and three act structure as part of the rules (the only other game that does it this clearly is the GM-less Fiasco, but most of the Powered by the Apocalypse families have character arcs built in).
A big part of this, however, is that most settings without shared authority are (at least to me) extremely unsatisfying for telling short stories because they skimp so badly on the first act and establishing normality. If we look at
The Matrix (to name a film I expect we've all seen) then Mr. Anderson's player knows in his bones what the office drudgery is like in a far more clear way than the GM can or will and thus the player should be carrying more of the load simply because although the character is not in control they probably know what will be said to them. If on the other hand neither the player nor the character knows what is going to happen those baseline scenes suddenly become interesting for the player. Likewise
The Force Awakens (for a more recent example), Rey's life scrabbling on Jakku is something she has a bone-deep understanding of and can tell a typical day of herself.
Perversely, although the characters become more powerful Neo
loses this understanding of his world when he takes the Red Pill - and Rey does when she takes off in the Falcon.
On the other hand you absolutely can play Sam & Dean's Endless Roadtrip Hunting Monsters without letting the players world build. This is because there is little relevant to their core activities that they both know at the start of the series and isn't covered by the rules. Xena: Warrior Princess works really well without any sort of shared authority.
Or in short Jessica Jones needs to understand her environment, and to do that it needs to work the way she thinks until it doesn't. Murdock and Nelson need to be able to deal with their jobs without asking the GM. When you try to do a miniseries and don't tie the character to the world you end up with Iron Fist.
Or in short, it's no fun to read a 'choose your own adventure' book when you are not only making the choices, but you are writing out what happens as a result. It's for those as yet unseen pages that you are reading and rereading the book. As soon as you know the book well enough that you know what each choice results in, you put it down.
When I listen to Act 2 of Hamilton or go to see a good performance of MacBeth I know exactly what is going to happen. This doesn't stop it moving me to tears. Mere discovery isn't the only reason to watch things or play RPGs and by crippling the understanding a character can have of their world you are simultaneously weakening the range of emotional engagement possible.
The story teller takes vicarious pleasure in their discovery and invents as needed, but he cannot discover anything for himself.
It sounds as if your group of players is simply boring. When my players were confronted by troglodytes having kidnapped a child to be eaten I thought they were just going to raid the camp and I was completely prepared for all the obvious ways I could do that. Instead they decided to dress all of them except the ranger up as emissaries of Blibbloppool, God of Troglodytes, and distract the Trogs while the ranger sneaked in the back.
The wheels may have fallen off that plan because none of them spoke Trog - but if you think I discovered nothing about the setting from that plan then I'm amazed.
The reader explores the story, but cannot create anything beyond his own unique narrative path through the story. Though that path, the reader has (hopefully) a grand adventure and experiences what would otherwise be impossible - living the life of an adventurer through that adventurer's eyes.
And here we come to one of your fundamental assumptions. The
only type of person I can think of who doesn't need a grounding in their setting is a professional adventurer. And yes, some fictional characters are adventurers (Sam & Dean, Xena & Gabrielle, etc.) but this is far from the only position.
If you try to violate that, no one at the table has that last experience.
Indeed. They have another experience because
not everyone wants to be a rootless adventurer. Being a rootless adventurer is far from the only way to play and is far from the most emotionally engaging way to play.
Apocalypse World has a couple of playbooks (notably the Gunlugger, the Brainer, and especially the Battlebabe) that are rootless adventurers. And more than a few that aren't for people who want to be embedded into the setting - and playbooks like the Maestro d', the Hardholder, and the Hocus get a lot of narrative authority over the setting. Not everyone always chooses rootless adventurers when given the choice and if you rule Bartertown then how Bartertown works is in part a reflection of you.
I often suggest that what makes a game fun is the illusion of success. That what any sort of game offers is the experience of being successful at something without nearly as much risk and hard work as real success has.
Once more you are working on the specific rootless adventurer power fantasy. When I'm playing
Montsegur 1244 I know I am going to fail. When I play Dread I know my odds of surviving the session (in character anyway) are genuinely low. This doesn't prevent these being engaging games.
Indeed one of the biggest issues regarding D&D in specific is that it contains not just the illusion of success but an almost complete lack of consequences. Characters don't get scarred. Characters are healed trivially easily and are as able on 1hp as they are on full hit points. Even death lacks its sting, and other than death the only consequences that can't be taken away with two nights of sleep and a few cure wounds spells and a lesser restoration are basically level drain and rust monster encounters.
When you say that what makes the game fun is the illusion of success a big part of this is because you are both good at what you do and another big part of it is because you fight tooth and nail against the mechanisms that make other forms of fun easier and more desirable. Mechanisms like grounding your characters in the setting and having competing factions that aren't fully under the GM's control so there are outcomes to fights that don't boil down to "Win or die".
It's generally a lot cooler when NPC falls in love as an emergent consequence of play
Agreed. But there are ways of making emergent consequences easier.
With Tabitha madly in love with him that became a major theme of the campaign, her love & devotion ended up reforming what had been conceived of as a pretty nasty PC, an outlaw who had accidentally killed his wife in a drunken rage.
