Lot of interesting stuff on this thread, but I'm going to go back to the OP's (hi, Darrin!) question:
How do the really great DMs out there manage to keep intricate campaigns with great story elements without making the players feel like they've been railroaded?
I tend to run pretty intricate campaigns and plot-oriented adventures. I think this is because I really enjoy them as a player, and, as a DM, one of the most satisfying moments comes with the big "reveal": the moment the players discover some key element of the story that makes them suddenly see everything in a new light. I don't know if I'm one of the "really great DMs," but judging from the longevity of my campaigns and the responses I've gotten from multiple groups over the years, I must be doing something more or less right.
So here are some of my thoughts on how to make story work without creating a sense of "railroading":
Establish clear context: Every story has a setting, and to one extent or another every story interacts with its setting. The more your players understand the setting, the more likely they are to behave predictably within it. (For example, if a player is conversant with Ebberon and knows the campaign is starting in Sharn, he's more likely to create a character concept that works in the Eberron setting and has a reason for being in Sharn than he would be if he was just told to make up a D&D character.) This predictability of behaviour is key, because it helps you write adventures that your players are likely to move through as anticipated.
Understand your story, but outline loosely: I find the three-act structure useful: In Act I, introduce or foreshadow the key elements that will drive the story, and end the act with the event that really sets things in motion. Build through the meat of the story in Act II, ending it with a major plot twist. The players' reaction to the plot twist leads to the climax in Act III. (This is just a loose guideline that helps me plan; my players are no more aware of these delineations than a movie audience or novel reader generally is.)
I work from an outline for the campaign, which notes events that need to occur in each adventure to make the story work. For example, I might need to introduce NPC X or reveal that Faction Y has a secret agenda in the fifth chapter of the arc. When I get to that adventure, I can design it based on what's currently going on in play, weaving in the key elements from my outline. My outline makes assumptions about the outcome of key plot points along the way, but only when it needs to and only in a very loose manner.
Build from what the players give you: I look for elements in the characters' backgrounds that I can build into the story. People, locations, and events the players come up with are usually easy to weave in to my own creations. This drives player involvement in the plot, which again makes it easier to anticipate their actions.
Some players take to providing backstory like a fish to water. Others really struggle. I have a number of techniques that make it easy for most players. For example, after about the fifth session or so, I ask the players to come up with "five things everybody knows about my character." (Even players who hate thinking and writing about characters can handle five bullet points.) Some will come up with stuff like "His hair is brown," while others will write detailed treatises on how their parents were abducted by demon worshippers. Both results are fine and help build character concept and a shared vision of the world, and I almost always get something interesting to build into the campaign out of each character.
Let yourself be surprised: Don't be a control freak. Things won't always go as you planned, but that's OK--in fact, it's great! I view unexpected player actions or results as a chance to revisit my story outline, often with even more inspiration and new hooks to build upon. In a recent session, a crit on a sneak attack with a high damage result killed an important NPC before he could (according to my plan) escape the scene. The players ended up with a mcguffin and some critical information well ahead of when I planned. No problem; I went back to my outline after the session and saw a number of possibilities for improving the story as a result.
I think a lot of "railroading" happens when the GM is too close to his original story idea--a great story-arc campaign has the GM's story at the helm, but is ultimately a collaborative effort.
Give everyone something to do: Understand what your players want out of their roleplaying session, and make sure everyone gets it. If you have action-oriented players, make sure you have an action scene or two every session, even in social/story/roleplaying portions of your arc. (And make sure the storyteller types get some plot advancement out of even the most combat-heavy sessions.) This keeps players from becoming frustrated, which in turn helps you understand and anticipate their actions.
For my current campaign, in which I'd be starting with a gaming group I'd never played with before, I wrote up a quiz (link in my footer) to help me understand what the players wanted before even starting the campaign.
Don't be afraid of "illusionism": I was unaware of this term before reading this thread, but I'm a strong proponent of the concept. Sometimes illusionism means "I intend for X to happen regardless of whether the heroes succeed or fail--but they don't need to know that." Sometimes it means "I have an encounter I'd like to run; I'll work it into the story when the time is right." Either way, it's the fine art of making it seem like the players' actions are driving events when, at times, it's really the demands of the story.
This is a fine art, but it's not as hard to do--and do well--as some people seem to think. When players are enjoying a story, they're more interested in story flow than whether a specific event appropriately rewards (or penalizes) them for their performance in an earlier encounter. As long as you keep things flowing logically and the players feel that they're progressing overall, it's not my experience that players want to analyze the results of a specific scene or encounter.
In my current campaign I recently had a large battle scene. I used the Victory Points system from
Heroes of Battle, and wrote up four possible results based on how many objective points the heroes achieved throughout the battle. To build tension through the session, I sealed the results into four envelopes, marked with the number of VPs needed to attain each result, and placed the envelopes on the table at the beginning of the battle. At the end of the session, I handed the players the appropriate envelope, then destroyed the others unopened.
Frankly, the players (to this day) have no idea of whether the other three envelopes contained the exact same results, or each something different. They had a good time, and the result they ended up with made sense. That's what mattered.
Build hooks early and often: Constantly introduce a rich tapestry of interesting NPCs, factions, and locations--especially in the first act. Later on, you'll see things develop in the story that you can hook back to these elements. This gives the story a powerful sense of continuity and depth. Plus, your players will think you're a genius when it turns out that some event from the beginning of the campaign is tightly connected to events at the climax.
Convert exposition into activity: In any involved story, you'll inevitably have some exposition. Straight exposition is boring in every storytelling form--especially roleplaying. So avoid it by putting it in the players' hands.
I started my current campaign with a one-shot adventure set about 20 years ahead of the campaign itself. It introduced the players to the setting, established a lot of the events that underlie the main story, and let the players get a taste of the homebrew rules I'm using before committing to their principal characters. I could have done all this with some big handout or a session's worth of talk, but it was much more fun this way and the players have a much more vivid sense of what set the campaign up.
Likewise, whenever there's a scene in which two NPCs talk to each other for more than a sentence or two each, I write up a script beforehand and print out two copies. When the scene comes up in play, I have two players take the NPCs' roles and read through the scene. It's kind of like a cut-scene in a video game, except more hands-on, and if any of the players want to interact, I can mediate as usual. It's way more vivid than having the players listen to me talk for ten minutes. (And the players think it's especially fun when one of them gets to play a major recurring NPC for a few minutes, even if they aren't making up the lines.)
I use exposition sparingly (the script technique above maybe twice in the last year of play), but I try to make it fun when I need it.
-----
This stuff is mostly about campaign plotting, but it all generally applies to individual adventures as well. These are the ideas I can come up with off the top of my head. Frankly, after 30 years of gaming, I'm still learning. Hope they help!