Tabletop roleplaying doesn't work if the referee is not a fair arbiter.
Keep in mind that the first tabletop roleplaying campaign out there Blackmoor didn't have a setup where the referee was running the "opposition". There were players playing the good guys and players playing the bad the guy. While Dave Arneson was running some NPCs, he mostly adjudicating between two opposing group of players. Later he did run the Blackmoor Dungeon where he handled the monster. But even then there were players who were involved like Sir Fang the Vampire Lord.
I am describing this bit of history to illustrate that the referee being a fair arbiter lies at the heart of tabletop roleplaying. WIthout it none of it work.
It now about having power. It about the fact that a major reason why tabletop roleplaying works is the players only know what their character knows.
Nor is dispersing the decision making about the setting, it locales, creatures and characters is a magic bullet for making a campaign better. Instead of relying on one guy getting it right, now you have to rely on the group getting it right. It can work, but small group dynamic ensures that there will be as many negative outcomes as there with a single human referee although they will be different.
I think you betray a very 'classical' notion of how RPGs work. This would have been quite fairly considered the consensus view, with perhaps only a very few dissenters, in something like 1980.
If you read Dungeon World, for example, nothing like 'fair arbiter' exists as a role. Nor is there a notion of 'secrets hidden from the players' (there may be things the GM has in mind that he will reveal later, IF doing so follows from the principles of play).
I don't disagree that giving players authorial power is no panacea. However, pretty much all modern 'narratively focused games' provide pretty strict regulation in that area. Again DW handily illustrates this, as it actually doesn't outright grant this authority to the players. Instead it provides pathways by which they can exercise it, and principles of play which govern the GM's process by which these pathways are actualized. There's no specific point in a PbtA game where the GM is told "the player is in charge of authoring content here" except in very circumscribed ways. Usually the GM is just told that he has to talk about a certain thing, explain a specific thing/subject/situation, etc. What I mean is, 'authorial chaos' would be a legitimate critique, but in practice it doesn't really arise due to game design techniques.
While your description of Blackmoor may be correct (I really have never read a detailed description of how it was played), oppositional play like that is a huge rarity in RPGs. It is more a feature of Free Kriegsspiel which fed into the 'Braunsteins' which inspired Dave's development of Blackmoor. I'd point out that this mode doesn't really work well unless you can either guarantee that both 'teams' are engaged in every session, or you develop a troupe play methodology. That may in fact be one reason that the LBBs were used that way a lot, by Gary Gygax for instance. Again, I don't know, but his early games sound like they at least featured competitive play, if not outright teams fighting it out with each other. Troupe play mostly died pretty early on, and I can frankly say I never witnessed oppositional play at all, and I started playing in about 1975. My point is, yes, a referee would be needed there, but not so much in modern play.