So you described what I call the random mess.
To move a plot or story forward in a meaningful way, events have to happen or not happen. You can't just have a random mess on top of a random mess. My way, and I think the best way, is for the DM to use Force to make things happen or not happen. Then I know of the scripted and Improv Quantum way......but what else is there?
This is why I gave the easy example: How to do a reacquiring villain? So, once Force is off the table, what else can a DM do?
I'm confused. Do you play in, or run games, where everyone just sits around the table saying "so what are we going to do in D&D tonight?"
The whole basis for this thread seems like a humble-brag.
"I hang out with a bunch of DMs and they all say how great my games are and everyone wants to be as great as I am, but I can't explain to them how to be as great as I am, woe is me."
I don't mean to sound harsh, but you described my answer, of how to "move the game forward" as you put it, as a "random mess."
I said "present the situations," I guess to you that means "force things on them." I think we're just using radically different language here, a real misalignment in communication.
But I'll try to answer your question.. "how to do a reoccurring villain."
Well, in my experience the first thing is to introduce the villain early. The players interact with them, and it's obvious through demonstration and description that the villain is beyond them- either because they're powerful, or because there are powerful consequences to harming/stopping them.
Have the players interact with minions of the villain, and have the minions either refer to or have commands from said villain.
If your question is "how do i have the players fight the villain but have the villain get away with out making it seem cheap," well, frankly if the villain gets away from the players when they're capable of defeating them it's going to feel "cheap" to the players because they wanted to kill the villain- but if it's done reasonably, feasibly, via spells or obvious magic items, then even though they might feel cheated they're going to hate that bastard even more and start thinking of ways to prevent them from getting away the second time. Counterspells, those manacles that prevent folk from teleporting, or a Forbiddance spell/trap.
Or you make it obvious that it's a one-time escape; "the ruby in their amulet shatters, enveloping them in magic and stealing them away." Next time they meet the villain, or see the villain, or scout the villain, they're going to ask "does he have another one of those amulets/rubies?"
Thirdly, you can make the character a foil, not a "villain"... at first. A great example of this is the Eberron dungeon magazines adventures featuring the famous Inquisitive. I know one of them is "Chimes at Midnight." There are several interactions with this famous inquisitive beating the party to the punch, or stealing credit from them, or being helpful but really annoying about it. Over the series of adventures, most players will really hate this guy but he's not actually doing anything wrong.
But then, he snaps and becomes a villain. And he's good at it. And the players finally get to take their frustrations out on him.