LOTS of great advice here. I can really only reiterate them and offer, perhaps, a couple of specific tricks of the trade.
The "don't plan too far ahead" is well-noted. Simultaneously, the having NPCs prepared and clear in your mind, whether the PCs are engaging them or NOT
So, I guess, my #1: Sometimes, what's going on "behind the scenes" that the PCs may or may not be aware of, is just as important [to the overall story] as what the PCs are actually doing/aware of.
That is...if there's some huge ritual going down at this particular night/time of year...the PCs may or may not have become aware of the plot. They might miss all of your carefully laid out clues. They might find the clues and come to a completely different conclusion...go traipsing halfway across the continent...in the wrong direction. As the DM you can a) change where the ritual is happening so they run into it anyway...I wouldn't, but you [as the DM] certainly could. Or b) scrap that piece of story/plot in lieu of something that the players think they figured out. (not an entirely bad option). Or c) come up with something entirely unrelated for the PCs to run into that does not effect the overall plot. Any way you choose, if it were me, that ritual should happen, on schedule, at the appointed time...with whatever outcome/consequences for the world [and overall plot].
#2: Along the lines of "knowing" your NPCs motivations and personalities...your villains, especially, play them SMART! Unless, of course, they are stupid lackeys or something. But the "big bad's", whether its their 5th level immediate threat, a 15th level regional enemy or the 25th level arch-nemesis who's been behind everything all along...These guys aren't [shouldn't be] stupid. ALWAYS...always, always, have
an escape plan ready for them. Did I mention ALWAYS?! A secret passage, a distraction to slip out the back door [i.e. they pull some hidden lever and rocks fall from the ceiling], magical movement spells or items (at higher levels "Contingency" is always great for this), a waiting enchanted hippogriff for them to dash to and fly away, whatever.
I have found over the years nothing will maintain interest and focus a party's aggression and thoughts [trying to "figure things out", which is what you want, right?] quite like 'hunting down that X bastard" who escaped them (preferably after doing at least some personal damage/harm).
Don't overdo it, of course. Not EVERY enemy can [or should] escape and the same enemy can't escape toooo many times before the party goes from "next time we'll get the bastard" to, "the DM's just being a d--k and saving his precious NPC". So it's a fine balance...but any villain worth their stripes is going to have an escape route/plan all set up...whether they make it there/get to use it or not. It should be there, in the plans/your notes.
{EDIT} Also, an additional thought on the topic of villains and lackeys...
A villain is only as good as his/her network. Sure, when the party is 30th level and they're going up against demon lords and lesser gods, the individual villain will/should be extremely difficult. But even Orcus will have unlimited undead at his disposal. Asmodeus has, literally, legions of devils to gate in. Lolth could/should have extremely high leveled drow and gigantic demonic spiders. The Fire giant king has his elite "honor guard" of giant warriors. At lower levels, the "Bandit King" has his gang or the mad cultist has his fanatical followers (probably a few spell casters included). And on and on.
The villain needs a "network." They will have a "right hand man/humanoid/demon/etc..." A yes man or two. Possibly a toady spellcaster. Then their "lieutenants" who each command a separate force of varying ability and force. Then the mass of soliders/minions/fodder. Taking on the "big bad" [whether 5th, 8th, 15th or 30th level] should not be just a "party vs. the individual"...unless, of course, the party has gone to great pains to catch/meet them alone, on purpose...and even then, help is likely an alarum away.
So thought/preparation/notes on those creatures/characters in place can be as important as the stats/prep on the "big bad" villain themselves. They don't all need to be fully fleshed out NPCs...though circumstances might bring the mid-level "lieutenant" into prominence in the story/make them a recurring foe. If this happens...and the party is then a few levels higher than their first meeting...the lieutenant should be as well. The PCs aren't the only ones gaining XP and increasing their power. To borrow from Dragonlance, suddenly facing Lord Soth [again] and/or taking him out becomes as important/pivotal as thwarting/facing Kitiara (or Takhisis, for that matter). Or from Hommlet/ToEE, taking out Lareth the Beautiful (who escaped when the party was 3rd level and is now/has become 10 levels higher than when the PCs first met him) becomes necessary to get to that planar portal to get to Lolth. etc...
I suppose, the point is, make sure the world setting is "living" and "breathing." The sun is rising and setting on the world, not just the PCs. The villain is organizing themselves, intelligent, and furthering their purposes at the same time the PCs are trying to figure out what those purposes are and/or stop them.
That's all for now...I'm sure the more I think on this, other stuff might pop up. But good luck!

[/EDIT]