How can we design scenarios and settings to enable players to find the facts on the ground and to provide them with enough levers in the fiction to interact with? What sorts of scenario and setting will generate an internal causality that is meaningfully knowable and comprehensible?
This is a really good question and I think far too many RPGs, historically, have been designed without considering these factors.
Spire: The City Must Fall is a great example of succeeding at what you're asking. Despite the setting being a miles-tall tower full of Drow being oppressed by High Elves, in a sort of 19th-century-ish setting full of weird magic and customs, it's immediately accessible to players because virtually everything in the Spire is conceptually analogous to real-world stuff, and the setting and scenarios are very carefully written to ensure that there are tons of levers accessible. I've never seen it done quite as well as this - the Sin sourcebook which just came out is truly spectacular in terms of levers, in that virtually every NPC and organisation is written in a way that it's going to be possible to manipulate it. The scenarios in Sin are likewise great.
A lot of that comes from writing flawed NPCs, flawed organisations, imperfect situations. Avoiding all-round badasses or organisations that are simply hypercompetent.
Re: causality, the setting is literally about change and achieving change, so the setting is designed to sustain and survive change in that you succeed in changing X here, it doesn't necessarily throw the entire setting off, and the consequences of achieving X are going to be fairly logical, and again the game makes sure to keep the GM looking at the consequences, like asking themselves "If the players are trying to do X, who will be trying to stop them? If X is achieved, how will people react? Who will try to take advantage, who will be having problems, and so on". Part of this means having a lot of factions which have somewhat overlapping functions. This is how the real world is, for the most part, especially outside of 20th and 21st century governments (but even they have a multiplicity of overlapping bureaucracies, and attempts to stop them overlapping tend to create gaps/cracks/holes people fall into - in some cases literally, with road management!).
So anyway I'd contrast Spire, which does it right, with Shadowrun, which for my money did it painfully wrong. Shadowrun, in virtually all editions, has huge problems with all three areas you describe.
1) It's much harder in Shadowrun for the players to find the "facts on the ground", because they're not where you expect them to be, and the setting is absolutely solid with people who don't want to tell you things, often would rather shoot you than give you the time of day (to a genuinely unrealistic-seeming degree), and the PCs aren't empowered to find those facts. Adventures/scenarios for Shadowrun often struggle hard to deliver the facts to the PCs. In at least a couple of cases they had a sort of primitive improvised "fail forwards" in that they essentially say to the DM "If PCs just fail at all this, have X NPC send them this!", which isn't er... great (it's not a proper fail forwards).
2) The hostile design of the setting and scenarios for Shadowrun, where the NPCs are typically described as either psychopathic, hypercompetent, or both, together the general "make them work for it" vibe creates a situation where it's very hard to find levers. The PC can, with effort, sometimes create levers, but there are typically few presented, and they often require very specialized setups or peculiar approaches (pixel-hunts) to discover.
3) On top of all of this, Shadowrun shares a common unfortunate trait with a lot of cyberpunk-set RPGs, in which the genuinely rational consequences of the game being played as described, will basically be that the PCs are rounded up and jailed for life, and/or "killed in a gun battle". That's just looking at the setting as presented, full of these hostile, vengeful groups, many of them with huge resources, and this digital world where, even if you're SINless, you're not actually that hard to track down. There's also the weird lack of rational consequence in the opposite direction, again common in cyberpunk RPGs - the PC are ludicrously underpaid for risk they're taking. Like by a factor of 10 or more typically, sometimes even by a factor of 100 or more. It doesn't make sense, rationally, within the setting that you could be paid that little. Sorry, I'm not here to "explain why we stole the cars" (the Shadowrun classic), but it is why! If you steal cars, the consequences are logical/rational. You get pennies on the dollar but the risk is controlled and limited and known and there probably isn't a firefight - yet even at pennies on the NuYen I should say, you're getting a reasonable amount.
So I guess my point with rational consequences is that they need to cut both ways, that PCs not getting what's coming to them, good or bad, undermines an RPG, rather severely. Don't write a setting as harsh and dangerous and Shadowrun and then design adventures/scenarios and describe the mode of play as if it's very much safer and less fatal! Don't expect PCs to risk death for less money than gets them to next week when that's clearly not how the world works.
And looking at '90s RPGs, it's wildly inconsistent - RPGs that are pulp as hell might do a reasonable job with rational consequences, but ones that are extremely elaborate and realist often fail miserably by the same token.
This is so obviously true that I'm disappointed you needed to say it. (And it needed to be said!) There are a lot of investigative scenarios for various games out there that explain in full to the GM what's going on, but so much of it is going to remain hidden to the players that I can't help but think, "What's the point?"
Whilst I definitely agree, there can be a point in a couple of ways, to secrets not necessarily revealed (imho of course!):
1) They can inform the rational consequences. I think this can be pretty important. In the real world, you rarely know everything, you rarely find out everything, and sometimes you just don't care, but for the consequences to seem rational, sometimes you need behind the scenes stuff going on, that the PCs don't necessarily delve into (or not in detail). I feel like, if the PCs always know absolutely everything about why what happened, happened, that's great for a murder mystery, but it's not so great for, well most other settings.
Of course you can always use film-like scenes to show undiscovered secrets if that makes sense in your game/RPG/scenario. I've done it before. Not often but it made an impact in a mystery/horror sort of game a while back.
2) Sometimes that stuff is there for the players if they want to dig into it. So it is wasted in the same sense that a dungeon with 20 rooms, where the players only visit 15 has wasted stuff.
3) You talk about levers earlier - and I think in the games I've been running lately, the largest "category" of "undiscovered secrets" is stuff I put there to act as levers. Like, the PCs might never find out X NPC was in love with Y NPC, because they just shot Y NPC in the face (to my surprise!), but that was in there to act as a lever, to make the NPCs and situation imperfect enough that it can be manipulated. Need that rough surface texture!
Generally this means there isn't going to be a whole lot that's not ever revealed though. I do know the kind of scenario you describe, often one where the PCs are essentially "lead by the nose" by NPCs/documents/etc. through the adventure, and never quite know what's going on, nor really could even if they cared to! That's not a great way to design things!
EDIT:
The TLDR is make the world and people in it imperfect and flawed, that'll give you the levers, and it'll help with the rational consequences -
@Umbran is I believe implying this when talking about how NPCs succeed/fail just like PCs. Avoid the hypercompetent.
Also when it comes to rational consequences, make sure the setting and the scenarios match up.
@Campbell points out how shocking twists rarely fit with good gameplay, and the same applies to setting and scenario design in other ways. The reason people always moan about pay in Shadowrun and similar games is that, intuitively, it's obviously nonsensical and unexpected that you'd be paid so little. That's an easy and tired example I admit but I think it helps make it obvious. Things should make sense, most of the time at least.
