I guess you could say my approach is to write a background story - there is a rough timeline of 'planned' events typically in a given region. I might have other things in play on a larger scale. They don't have to be heavily details, and the timeline can easily change, especially based on what the PCs do. Based on your description, you have some of this already, a few maps, some stat blocks, etc. I guess the major difference is that I also have some active stories. If you're familiar with FR publications, this would be the 'current clack' section. Other rumors and activities may be developed on the fly.
I guess that the reality is that there is a continuum of prepared and random/on-the-fly things all of the time. A published adventure tends to lean heavily on the pre-planned. At the very least there's an intended end-game. I'm comfortable more in the middle. I like to think through things a bit more (duh) so I have a lot of semi-pre-planned activities going on in my head, and sometimes on (digital) paper to give me a head start in the event that the PCs intersect that story. After each session I typically have a bunch of ideas in reaction to the session. As that percolates over the week it tends to coalesce into a sort of 'prep' for the upcoming session, although it may never end up written down.
But I always have a pretty good idea of the various directions things can go if they stick with the current goal. If they change their goal (which they have done quite frequently), then it gets interesting. But that's also why I have a good idea of most of what's going on in the background so I can react quickly and consistently.
Ilbranteloth
First off, the way you've described your pre-campaign setup is similar to mine. I think it's important for the characters to have a reasonably clear picture of how their characters are "framed" in the fiction at the get go. Some people advocate a 100% "No Myth" approach (meaning the absolute bare minimum of backstory), but I've found that to frame relevant conflicts and scenes, I have to have a reasonably good grasp of what's happening in the game world.
I agree that it's important to have a "semi-breathing" world, where there are NPCs, organizations, nations, etc. that have agendas and will be taking actions in the background whether the PCs adapt to them or not. What shouldn't happen (at least in my opinion), is there shouldn't be some final encounter, some end game or plot development that WILL INEVITABLY HAPPEN, no matter what.
My GM-ing took a dramatic leap forward when my focus became about engaging the PCs with what was interesting to the players. To that end, scene frames, situations and NPC motivations have to remain adaptable and fluid. It was about shifting gears and changing directions based on what you see at the table.
What I don't understand is, if everything was randomly determined, and particularly if the location of a given object, person, etc. is partially dependent upon the success or failure of a roll, how can you develop a campaign world with depth and consistency if you as the DM don't know stuff about it beforehand? I have potential antagonists going about their plans whether the PCs intervene or not. Even if they don't directly engage with them, their actions can still have an impact.
So I guess this is where it starts to lose me. Depending on the circumstance, I'd agree, not everything is determined ahead of time. There are some random events, encounters, NPCs, etc. I'm OK with that as part of the basic framework of the world. But I also view it as a living, breathing world that needs consistency. When you start with a clean slate, never adventured in the world before. Fair enough. But once those NPCs have been met, they exist. If they run into them again, there needs to be consistency. I guess if you're just keeping track of things as you build it you'll end up in the same place, but I get the feeling that more involved, deeper plots are difficult to pull together in that approach.
I think I understand why you're getting hung up on this point, but the trick to remember is, as a GM you're not authoring "in the moment" all the time. There are absolutely moments where your pre-authored fiction can and should hold true. Sometimes the mace isn't there, and it doesn't matter what the PCs do, or say, or roll on the dice, they ain't gettin' that mace in Location X.
BUT --- and this is the big "but" --- as a GM, you should always be asking yourself, "Is that REALLY the case? Does the game get any better (or worse) if I decide right now, in the moment, that in fact the mace is there? And that the PCs will find it?"
If that's the case, don't even make them roll for it. There's no roll to be made. Assuming the PCs declare that they make any kind of reasonable search of the area to find the mace, they find it. This is classic "Say yes or roll the dice." Well, we just said yes --- the mace is there. Now what happens?
Or if they do roll for it, decide right then and there that they will find SOMETHING, even if the check result says "failure." This is the idea behind "fail forward." Sometimes, the PC's success is guaranteed, but I still make them roll the check to see how long it takes, the degree of success (which can lead to more interesting findings), and whether anything in the scene around them reacts to it.
To me the whole "Climbing Mount Pudding" example is a classic case of process sim run amok. If it's so dang important for the PCs to make it to the top of Mount Pudding, then scene frame it such that the PCs make it to Mount Pudding, but use the climb checks to represent some other variable other than the success of the climb.
Oh sure, if you change your pre-authored fiction, you're probably going to have to change other stuff in the fiction too, and maybe even do a mild ret-con (though in my experience, even if it's a semi-obvious ret-con to the players, most of the time they don't care and just roll with it). But if changing your pre-authored fiction increases the dramatic tension, pushes narrative momentum, and gives the PCs a chance to really dig in further to their character goals.......isn't that BETTER?
(Obviously the hardcore "simulationists" will disagree, but frankly I don't care. In my experience the only time "simulationism" works in the first place is if the players are heavily invested in their PCs with goals, motivations, and back stories . . . in which case, why would I purposefully use heavy process sim / pre-authoring to stunt their ability to engage with what they want?)