. As a DM this is something I try to embrace. As a player, I don't like to be 'forced' down a path either, even though I understand it's 'the story arc.' A really artful DM can plant a plot hook and bait it so the players want to follow it. Do I claim 100% success with this approach? No, but at times, yes. The flip side of this coin is the 'contrary' player whose enjoyment in the game derives in spoiling / disregarding, or DERAILING(!!) all the preparation the DM has put in beforehand. Frankly, my toleration for this is pretty low.
As a long time DM, I will throw out hooks (note the plural). However, I never have expectations players will bite on any of them. On occassion, I may place the characters in a situtation (e.g. In one campaign, the party was repeatedly ambushed by members of a wizard's guild they crossed during the first two sessions. Unbeknown to the party, the rogue had pocketed a ring that he took off a dead guild member's - the ring was enchanted by a paranoid guildmaster whom enchanted the rings to track the movements of the other members).
(I've been playing since 6th grade in 1979) After all, you do other things with your friends and family, and the gaming table isn't your sole focus of association with them. For that reason, problems and problem-play can more easily be addressed and corrected, and to an extent, you can see (and sometimes even participate in) the effort it takes when someone you know / associate with other than at the game table is expending large amounts of effort in adventure / world building. In the most recent campaign that I ran, I put in over 40 hours of prep time for what was going to amount to about 12-16 hours of play time (about 4-5 sessions)- part of the reason it was so extensive was that it was an urban adventure (requested by the players after a few sessions of dungeon crawling) with a ton of interconnected but separate plots / mini-adventures, specifically to give them some of that 'sand boxing' feel, even though the whole thing was set in and around a single city, and was therefore in-and-of-itself a form of 'railroad', right? For players who have never run a game, stop and think about that for a moment about what would happen if they all showed up and said 'F that. We're going to leave the city and head back to the forest. And by the way, you suck as a DM.' That's the sort of thing that sends DM's looking for new players, or to say 'screw it' and move to the players' side of the screen permanently. I'm just sayin'.
Well, I started playing the same year that you did and began DMing the following year. The players may have asked for an urban adventure, but I think you may have spent too much time planning multiple adventures all at once and became too invested in
your story and plans (based on what you wrote). There is nothing wrong with players changing their minds or going off on a tangent mid-adventure, It happens.
Until around 1985, I was focused on the dungeon of the week. There would be the occassional town (including using T1: The Villlage of Hommlet and L1: The Secret of Bone Hill). However, in 1985, two DMs (well, Rolemsaster GMs) introduced me to a different style of prep. Since then, I don't begin with any story in mind nor do I do any adventure prep until I have the initial characters, their backgrounds, and goals.
The majority of my initial prep, has since, been about world building and creating a house rule document - both of which are done prior to character creation. The world establishes a map of the setting (with notable geographic features), cosmology (e.g. what other planes exist), deities (if they exist), the existing races and cultures (e.g. notes on subsistance, political organization, social stratification, technology, belief systems, social mores, available classes and subclasses found within), the nations, major cities, major organizations (e.g. priesthoods), major NPCs (and their goals), a little local history and some current events (that may serve as hooks for backgrounds and adventures).
As for the initial adventure, I try to use the characer backgrounds and goals as the basis for the adventure and to bring the characters to the starting location (typically, I also throw, in an initiating event). I don't over prepare the adventure. I have a general area map, general maps of any important locations in the area, and any relevant notes for important NPCs, typical mooks, monsters, and magic items.
After the intial adventure, I follow the PCs. Do they have a specific location where they want to go based on their goals or backgrounds? Did they grab a hook?
Along the way, they may find themselves stumbling into situations based on where they are in the world. Enemies they have made may appear (e.g. as happened as a result of the rogue and the ring). They may encounter new hooks (e.g. they may hear a rumor, someone may request help, etc.) which they are free to ignore (however, ignoring such requests may have its own consequences done the road). Again, I'll have general maps and notes prepared just as with the orignal adventure
Players may even come up with new short term goals (e.g. in a campaign, the party arrived in a city and tried setting the priest (i.e. druid) up with a prosititute because they found him to be to rigid).
By keeping things loose, I don't let myself get tied to a particular "story" arc or adventure. If the players abandon what I have planned, that is fine. If we are not near the end of the session time, I ask for a short break and write up some notes drawing upon what I prepared in my pre-campaign prep (and, possibly, from prior in game events) to help me.
For example, three years into a campaign (in which half the players were also GMs), I had one session in which the PCs decided to explore the ruins of a banished Necromancer's tower. During the exploration, they found the remnants of his journal (much of it was burned). A partial passage (meant as piece of history) caught the attention of the players and, based upon a prior session's events, they decided that someone might be trying to bring the Necromancer back into the world. As a result, they chose to abandon the ruins that I prepped for the evening. Most chose to seek out a wizard whom was also one of the two great sages and had banished the necromancer over one hundred years ago. Two wanted to return to the town where, months ago, they had found some people suffering from a disease and zombies rising from the dead on their prior visit
I could have told the players that they had to stay, because a) this was one of the few actual dungeons that I had created in ten years; b) I had fleshed it out more than normal; c) they were going off on a tangent (sort of, as someone was attempting to bring the Necromancer, but it had nothing to do with the journal (see below)); and d) I was not prepared for them to go to the wizard- especially with this information. I could have also told that players that they cannot split the party. However, the game is about the PC including their decision to abandon adventures so I went with it
I knew a few about the wizard. First, he was one of the few non-guild wizards not killed by the wizard's guild; 2) he had monsters along the way to his tower to discourage the average person from bothering him and to discourage the wizard's guild from attacking him; 3) he, not the Necromancer, was the villain in the historical event. The Necromancer had tried to stop him. However, based on his appearance, some prior actions, and that he worked with necromancy, people would not believe he was the good guy (in this instance);
I asked for a break to generate some encounters for both groups. One group made it to the wizard. To keep up appearances, he managed to convince the first group to go on a quest to stop the Necromancer from returning which just coincided with one character's background to find his sister and her adventuring party (They disappeared in a prior campaign trying to bring back the Wizard as they had thought the Wizard was the Necromancer in disguise and the actual Wizard was banished).
The other group, having to travel farther, found that the disease and zombies were spreading.
When the group met back up, the next session, to my surprise, they did not go after the Necromancer as planned. They decided to split up into four (or was it five?) groups to tell their homelands to prepare for the disease, armires of zombies, and, potential, war against the Necromancer.
My notes for the session "went out the window" again. It was ok and I dont think the players were jerks.