I agree. I just like tightly crafted stories. They are more interesting and more challenging. Nothing bothers me more as a player than going into a battle that you can tell the DM just came up with on the fly. A battle with some goblins on a flat plane with no terrain is no fun. A battle with goblins who are hiding in trees with a leader than enhances the rest of their abilities, with an artillery hiding behind cover while the soldiers stand up front is much more fun.
That sort of thing CAN be done on the fly, but it isn't easy to make it work with very little notice.
I consider being able to run on the fly an essential skill to running a good game. I have three main ways I do this. First, I will make house rules or set up tools to allow me to throw things together quickly. For my Hollow Earth expedition game I put together a spreadsheet with the basic stats for most of the adversaries in the book. Not everything, but enough to get me by in a pinch. Pick a bad guy, think of a cool scene, and see what they do with it. For example, I glanced through and saw Ape Men, and thought an oasis with a tribe of these ape-men would make a good encounter. The PCs hadn't seen them before, and it was a little different than the dinosaurs and vikings they'd been dealing with.
Second, I think of what the PCs are going to want for a cool encounter. In that same scenario, two of my PCs have a motivation of Duty, so they try to protect the innocent and so on. One of the ape-men is tied up in front on a large cave, while the others are gathering fruit. They're peaceful, and offering a sacrifice so they can use the oasis. The PCs figure this out, and go to free the poor beast. Out comes - well glancing at my sheet a Giant Ape. King Kong comes out and we have a really cool fight. Terrain? We've got PCs hiding in the cover of the trees, and the ape climbing up the mountain like its the Empire state building.
Finally, I make healthy re-use of the stuff I had prepared. In that game I went signifigantly off-script. They needed a whosits from the captain to get a McGuffin, and I'd assumed they'd have the spy try and seduce it out of him and had some fun encounters planned in the meantime. Instead, as soon as they got there they pulled a gun on the captain and demanded it from him. Which, being a pirate, he kind of respected. So I didn't get to tell the scholar about the lost city, or have the Overconfident character face off against a saber-toothed tiger to prove her courage. But, they got on the trail a lot faster than I'd planned. I figured the whole game to be in the pirate cove, and they were done with that in an hour. So they got marching and I put the ruined city in their path for the scholar to explore. While exploring, they were stalked by a saber-toothed tiger.
I see no reason you can't do the same in a D&D game. So they don't want to go into the Caves of the Unknown to face the BBEG and all the encounters you prepped. So, reuse them! And there's nothing wrong with pointing them back. You had a cool encounter with goblins using pits they can push PCs into, and they decide to explore around the mountain instead? Pop that encounter into the woods, with pits dug in the forest instead of the dungeon. Maybe the leader has some orders that they're to report any strangers back to the tribe in the Caves, because the Boss wants to know if anyone is snooping around.
I also like stories that make sense. If the BBEG doesn't even know the town they are in exists and they decide not to head into the Caves like I expected, then simply nothing happens. They spend their lives for the next couple of years doing nothing interesting in the town. They don't have any adventures. If they decide to head around the mountain instead of going in, they don't gain the clue or levels I expected them to, they never figure out the clue they need to defeat the BBEG and they fail. I leave the choice in my player's hands, but there often is a choice that is "the best".
This is almost word for word the description of a type of railroad in the 2e DMG, if memory serves. So there's nothing going on in your world other than what you prepare? What kind of BBEG is this any way? Don't they generally have Evil Plots Afoot? If they don't go into the caves, then the city starts seeing weird cultists, or orc raiding parties, or whatever kind of nastiness he's up to.
The clue problem can be an issue. I run a lot of Cthulhu, where the PCs are generally doing a lot of clues and information so they can figure out what is going on. If they miss a vital clue, the game could grind to a halt. So I deal with it by making a clue net. There's something going on, and there's a bit of it here, bit of it there, and if they miss one they might lose out on some detail but they can generally figure out what's what.
For me, running on the fly is one of the greatest thrills. Once you get some good tools set up and some confidence, its really not too hard. These days I never try to protect a plot, I don't even write them. Throw a situation at them, and see what they do with it. Having a 'best' answer is boring. Having several answers all with their own moral and practical implications makes for awesome gaming. IMO of course.