My basic issue with sandboxing is generally pacing. Which, to me, is exactly what the Thief VS the King example represents. It's not that stealing from the king is bad or good, it's that it can very, very quickly grind the game to a complete standstill while we attempt to resolve what happened.
Laying aside the socially dysfunctional people you say you play with for a moment, let's focus on this: Your entire position relies on the assumption that the GM's plot is more interesting than what the players want to do.
Sure, you can't choose to do X, Y or Z, but, you will progress through things much faster, so, it's not boring.
Progress through
what? Your claim relies entirely on the premise that the only thing that matters is ticking off items on the GM's pre-planned checklist. If the checklist doesn't exist your position becomes fairly incoherent.
If we're talking about the speed with which something exciting happens, then the thief pick-pocketing the king has just mainlined that process. No matter how that action resolution turns out, something exciting is about to happen:
- You're going to get something valuable from the king.
- You're going to get involved in an epic chase through the palace.
- You're going to be arrested and thrown in prison and need to concoct your escape.
- You're going to be captured and sentenced to clear out Norworld and make it habitable for civilized folks.
I think the pre-determination that any of this is less interesting or slower paced than "have a chat with the king where he offers you a contract to go clear out Norworld" is to make a rather large presumption that the GM's predetermined plot is the only interesting outcome.
The "wing it" element of running a status quo setting may include going off the edges of the map, but more importantly, in my experience, it's about the setting reacting to what the adventurers are doing. Frex, if the adventurers take sides in a dispute, that should have ramifications; the referee responds in character- and institutionally-appropriate ways to the actions of the adventurers - it may attract new allies, it may earn them new enemies, it may bring them to the attention of the authorities, and so on.
QFT.
Let me give another example of how smart prep in sandbox play means less prep and easier prep than for plotted adventures.
In my current campaign I prepped a "backdrop" in which the Dragon Church would fracture into two competing factions. (A "backdrop" is a sequence of events which the PCs will hear about through rumors, newsheets, and the like.) These events in the Dragon Church were designed to establish certain elements of the church which would factor into a seemingly unrelated sequence of events that one of my PCs had gotten deeply involved with.
When one of the PCs decided to attend a local tournament, I included the order of knighthood associated with the Dragon Church among the orders competing in the tournament. At this point I was able to reuse the heraldry I had designed for the upcoming backdrop of the church schism.
A little while later, this same PC decided they wanted to pursue a knighthood. Recalling the tourney, they chose the order of knighthood associated with the Dragon Church and approached it. Their application to squire themselves was accepted.
Now that the Dragon Church knighthood had been more immediately engaged, I designed three stat blocks: One for the leaders of the knighthood, one for elite members of the knighthood, and one for the weakest members of the knighthood. I also fleshed out my notes for three NPCs (the PC's trainer, the head of the order, and the order's second-in-command) and gave each of them a stat block (elite, leader, leader).
Total prep so-far: Graphic for the heraldry. One page of notes on the church's doctrine. Three stat blocks. One page of schism events each described in 1-2 sentences. 2 pages describing the major NPCs involved.
Here's where the sandbox really kicks it into gear:
(1) The first schism event kicks off with an assassination attempt on the head of the order while he's at the tourney field. This was literally three sentences of prep: "Abanar takes several knights to the tournament field to recruit them in opposing the False Pope. Sir Gemmell, under orders from the False Pope, sends two men to assassinate Abanar. Abanar's knights kill the assassins, they ride to the Godskeep, and are driven back by Gemmell's knights."
But through sheer coincidence the PC squire decided to visit the tourney field that day for completely unrelated reasons. The PC ended up helping to thwart the assassination; rode through the streets with Abanar's knights; helped to assault the Godskeep; and then escaped during the ensuing rout.
(2) The PC was then approached by both factions in the dispute. After several intense roleplaying encounters he chose Abanar's side.
(3) The PC then inadvertently betrayed Abanar's location to Gemmell through another series of accidental coincidences that led him to believe (much to my surprise!) that he was facing a test of loyalty. He and the other PCs then narrowly managed to rescue Abanar. (Used those knight stat blocks again.)
(4) They secured Abanar in an abandoned dungeon complex they had cleared out several levels earlier. (More recycling of material.)
(5) The PC is now serving under Gemmell while secretly spying for Abanar.
So from about 5 pages worth of prep, we've generated several sessions worth of exciting play.
And, ultimately, what's the secret behind the success of this improv? Is it because I'm amazingly awesome?
Nah. It's just (a) prepping raw material with dramatic potential; (b) opportunistically empowering the players; and (c) roleplaying.
I'm not succeeding in the absence of prep. I'm succeeding because I'm prepping the right stuff and not wasting my time on less effective prep (like predetermined plots).