The question is how do the really great DMs out there manage to keep intricate campaigns with great story elements without making the players feel like they've been railroaded?
Ah, this is really the key, I'd say. You can have a plot that runs on big ol' rails, but you have to make sure the train has shock absorbers so the PCs never feel them.
IMO, the most important thing is to get the players to buy in to your plotline. Once they're invested in reaching a particular goal, it becomes vastly easier to keep things on track; all you have to do is show them a path leading toward that goal, and they're very likely to follow it. But you have to recognize that you can't force them to buy in. If they don't like your chosen plot, you're gonna have to scrap it and make a new one.
I generally start off with a giant obvious plot hook. If something about a PC's background piques my interest, or if the group has expressed interest in a particular type of campaign, I'll build off that. Otherwise, it will probably be a standard "save your little chunk of the world" quest. Usually the PCs take the hook. If they don't, okay, I'll improvise something and we'll spend the rest of that session futzing around. Then I'll come back next session with a new plot hook. Repeat until the players bite on one.
Alternatively, the PCs may take it into their heads to undertake some epic task all by themselves, without any plot hook provided by me. That's cool too - in fact, it's better, because players who've set their own goal are usually extra committed to it. I'll improvise through the session, then sit down afterward and work out a plot based on whatever it is they've decided to do.
Once the PCs have bought into the plot, then it's just a matter of maintaining forward motion and keeping them from getting bored. I give them milestones (not in the 4E rules sense, but in the sense of "a series of smaller goals on the way to the big one") so they can tell they're making progress, and try to make sure they always have a sense of what they need to accomplish next. I mix up combat, social, and exploration encounters so they don't fall into a rut, and try to make sure every encounter has something new and different going on so it doesn't become a slog. Nothing sends PCs off the rails faster than boredom.
When designing encounters, remember that PCs are notorious for talking to folks you expected them to attack, and attacking folks you expected them to talk to. Plan for both contingencies and make sure neither is going to wreck your plot.
Above all, be flexible and don't sweat the small stuff. The PCs do not have to do everything the way you imagined them doing it. If they come up with an unexpected way to reach a particular milestone, roll with it; make them work for it but let them have it in the end. If they abandon your milestones altogether and go in a totally unexpected direction, just make stuff up for the rest of the session, then sit down afterward and figure out a new way to reach the objective from the PCs' current location.
A question about the meaning of "railroading": in most of the posts above, it seems that "not railroading" means letting the PCs go where the players want them to go in the gameworld regardless of the GM's preference.
But is this sort of sandbox the only alternative to railroading? As I indicated in my post above I don't think so, but I'm curious about what anyone else thinks.
It's more or less a given in my games that the PCs can go anywhere in the game-world the players want to go. My job is to make sure that the place they want to go is the place where the plot rails run. Usually that means leading the PCs along the rails. Sometimes it means ripping up the rails and laying them back down in whatever direction the PCs are heading.
