Let me add in my $0.02 with some examples, though it won't be much different than the already great advice presented here.
My DMing philosophy is generally that "the DM builds the world, but the players build the story by their actions within in." All I do is set the possibilities into motion. I come up with a few basic motivations ("Evil Group X wants to achive Y, and will be doing A, B, and C"), add in a few elements from the player's character background ("Player H hates undead, so lo, there are many undead to be hunted in the Barrowdowns"), and add some general staus quo encounters ("No one has ever returned from the Dungeon of Death!"), then turn the players loose to act on their own motivations.
Now, this means the players can go anywhere, and might surprise you, so how do you keep it under control? I use two techniques --"Current Clack", and. modular elements and adventures. I usually have a running list of rumors that the players have access to when the Gather Information upon entering a town or village, or conversing with an NPC -- about ten or so at anyu given time. Maybe two will be complete red herrings, two or three will be bits of local news that the players generally find dull (Farmer Jones has a new cow) but could later become something interesting (But it's really a demon cow in disguise!), and the rest point toward the primary adventure elements I have ready - -which are usually tied to different (though potentially linked) plotlines that I came up wityh early on. That way there's always two or three major options for characters to pursue, several of which are directly tied to their motivations, so I don't get surprised too often. I try to keep in mind adventures I'd like to run down the road, and begin dropping hints to those early. Sometimes they come into play, and sometimes not. I also update previous rumors that the party has heard over time, so some adventure hooks gradually disappear, or are taken care of by NPC adventurers (who are often parties the PCs have met, and might have a friendly or unfriendly rivalry with).
How do I prepare all those options without going crazy, or spending lots of long hours in preparation? Easy -- modules. I use a lot of preprublished adventures from all sources -- old Basic, 1E, and 2E adventures, 3E adventure path material, Dungeon magazine, d20 PDF and print publishers, free material from the Internet (you can build a huge library of adventure material just by downloading free stuff from the 'net). Now, I generally tweak those products to fit my specifics -- cahnging a few monsters; a key NPC might become one the PCs have encountered before, or an evil organization might become a campaign appropriate power group -- but I find that's much easier than creating a lot of stuff from scratch. I do create my own original material, to fill niches that aren't filled elsewhere, or provide link between other adventures, but I try to focus my time on the structure of the world (how the elements fit together) and make as much use of other peoples' stats, mechanics, and maps as possible.
Now, every now and then players will surprise me, and chase a red herring, or develop a burning desire to do something I hadn't prepared for. Here are some of my tricks for being prepared for that:
- I keep a blank flowchart in my DM notebook, that could be a chase scene, a wilderness exploration, or a random dungeon. This came in handy when the party decided to venture into the city sewers chasing a bad guy (and I hadn't prepared the sewers in advance).
- I keep a few lists of random NPCs and adventuring parties they can encounter (caravans, wandering merchants, and equally-matched party of competing adventurers). This way there's someone they can meet on the road, or talk to at an inn, to provide guidance or competition.
- I have a list of encounters that fit well for travelling, for when I need to shift gears, or change the pace of the adventure. Think of it as a wandering monster table, but it isn't random.
- I have a couple of generic "Side Trek"-type adventures I can stick in anywhere when the mood strikes or I need to tweak events -- usually a few Dungeon Mags thrown in my bag, plus whatever good idea I'd come up with lately for a small encounter that doesn't have a firm place in the rumor mill (like a raided, burned caravan that could lead to a search for bandits, or an ambush by xenophobic elves, or something).
Here's an example of how I put all that together, from the start of our current campaign over a year ago (you can check out the first page of
diaglo's Story Hour to see how it turned out):
I decided to set the campaign around a small town (conveniently pre-statted from the FRCS, but without so much detail that it was easy to change things). The area has been raided by non-human monsters and some other forces, so I established a few gerneal goals for the various groups, then decided what their lairs would be based on some of the adventure material I had available (selecting the <i>Keep on the Borderlands</i>, <i>Sunless Citadel</i>, and "Raiders of Galath's Roost" from <i>Dungeon</i> as the starting options). I decided that running other Adventure Path modules might be a possibility down the road, so the general background information provided to the players included the information that the local dwarves had once had another home they were driven out of (Hint #1 for <I>Forge of Fury</i>) and that the area had once been raided by an enourmous red dragon (which could be a link to a number of different adventures).
Starting the campaign, the party of 1st level characters met and organized, and encountered a couple of original encounters I'd prepared to set the stage, at which point it was revealed that three major events had occured that the locals were concerned about (1) A little girl had been kidnapped by goblin-like creatures (<I>Sunless Citadel</i> hook), (2) A farm to the north was raided and a patrol of militia disappeared ("Galath's Roost" hook), and (3) caravans to a settlement to the south had been raided by goblins & hobgoblins (<I>Keep on the Borderlands</i> hook). Now, I didn't really want to run <i>Sunless Citadel</i> much, so I weighted the hooks a bit by offering a small fee to escort a new caravan southward. The party could take up any option -- but they chose to go south. I then closed the <i>Sunless Citadel</i> option by having another adventuring group take up that particular quest, and every time the party would return to town they'd hear about a missing party of adventurers, then that one lone adventurer had returned with the missing girl, but that a famous paladin was still missing ... (the players who were familiar with the adventure eventiually picked up the cues, and enjoyed the ongoing mystery about the wherabouts of Sir Bradford, even though they had no intention of running of to rescue him, though they certainly could have).
Regardless of which way the party went, I had a side trek encounter ready for them -- a toll bridge manned by a group of raiders that would lead to a raided farmhouse -- that I'd fit to the particular situation by changing the creatures and the hints left behind in their gear (also dropping in a masterwork weapon with the mark of a long-lost dwarven smith, <i>Forge of Fury</i> hint #2). The party eventually arrived at their desitnation and pursued their adventures; while they were doing that I was setting up future possibilities. Each time they returned to town they would hear a few different rumors; those plus a few escaped bad guys motivated their continuing adventures. They eventually did take the <i>Forge of Fury</i> hooks, but that's a story for another day ...
I've integrated a wide range of material using this technique, and everyone's seemed to have fun, and it really hasn't been that much work to do.
Good luck!