Probably a bunch of this is covered by other posters above, but I'll give this a shot. I haven't done an full level 1-16ish AP per se, although I've played and DMed some fairly long adventures. I think most APs work through a combination of player buy-in, explicit bread crumb trails, metagaming, and limiting information and resources.
First off, what do I think an AP involves? An AP is a series of scenarios that the players engage in sequence. Each scenario is basically a mini-adventure, maybe enough for 0.5 to 2.5 levels. The scenarios are often site-based, but can be event-based. So for each scenario the AP should set out the personality and resources of the NPCs. For each scenario, the goal is implicit from the AP setup, or is given by a quest-giver. Its up to the PCs on how to accomplish the goal, and the AP gives enough information for the DM to adjudicate the most likely techniques. The DM is expected to adjudicate unexpected actions by the PCs based on the personality and resources of the NPCs.
Example: In the starting scenario, a friendly NPC asks the PCs to stop raids by goblins. The AP has a map of the goblin caves and sets out enough info to handle PCs deciding to kill the goblins or bribe them to go somewhere else. In the second scenario, a band of ogres will threaten the village. The AP has a map of the fort used by the ogres, and sets out enough info to handle PCs deciding to kill the ogres or trick them into leaving.
Player buy-in. So, as a player, when I start an AP, my understanding is that unlike a sandbox where I'm driving the overall goals, I'll accept the goals set out in the adventure path. I expect that I'll either have a quest-giver as a starting condition of the adventure or encounter one early in the adventure. I know that the quest-giver often has own goals and may even eventually be an enemy, so I'll be alert to the possibility of being a patsy, but to get the ball rolling or keep it rolling I'll accept the task.
Example: The DM says "Hey guys, I'd like to DM Rise of the Runelords adventure path. It's sort of an homage to the G series, with fights against goblins, ogres and giants. Are you interested?" Players "Sure, sounds like we need multiple rangers and clerics."
Bread crumbs. Either there's fairly clear clue to the next scenario is placed at the end of the current scenario, or the quest-giver has a new task.
Example: If the PCs negotiate with the goblin chief, he says he's attacking based on orders from ogres in the West Hills. If the PCs kill the goblins and search, they find a crude letter in goblin from the ogre leader. An NPC will be able to read the language of the letter if none of the players can. If the PCs kill the goblins and burn everything to the ground or don't search, well in a few weeks a couple ogres will attack the village, and the quest-giver will ask the PCs to deal with it if they don't decide to on their own.
Metagaming. Here I think of two factors. First, there are some possible solutions to the goal that players are likely to reject based on the understanding that they are in an adventure path. Could the PCs suggest that in response to the goblins menace, the whole town should simply up and move away? Sure, it's possible. But as a player I generally expect that the AP will have the bread crumb trail, and the crumbs will start where the goblins are. So my initial choice would be to interact with the goblins somehow. Second, if we're talking 3e style adventure going from level 1 to high level, there's a pretty steep power disparity. So if I get rumors of ogres in the west hills from an NPC when I'm first level, even if I suspect there's a connection, I'm not going to investigate that yet.
Limited world-based resources. So, I think APs work when the PCs start at low level with limited magic and players don't hold a lot of information about the rest of the world and/or powerful NPCs are reasonably far away. The more well developed the world into which you attempt to slot an AP, the more likely you'll have to deal with unexpected PC actions, particularly at higher level. In a well developed campaign, you might have PCs saying "Hey, remember that white dragon we encountered, I bet it hates efreeti. It might help us." Or in Forgotten Realms, you have "Let's go ask Elminster for help." The AP assumes that starting PCs just don't have that level of knowledge and that Elminster is far away or is busy. Now, if the PCs encounter a white dragon as part of the adventure path, and later think of getting as an ally? If I'm the DM, I think that's great! If the AP doesn't address that, I'm going to adjudicate based on the personality and goals of that dragon.
Example: The village being attacked by goblins is in a generally remote area, far from large cities or any standing army of the nation, so that the PCs have to deal with the situation themselves rather than go to authorities and ask for a squad of soldiers to be sent.
Limited information. Here I mean that in order to prevent the players from jumping ahead, information about the future stages simply isn't revealed until late in a given scenario.
Example: After defeating the ogres, the PCs find out that they are under orders from hill giants. During a raid on the stedding of the hill giant chief, they encounter a frost giant embassador. Even if players know drow exist, do they have any reason to venture into the underdark?
So, back to the questions. What keeps the players on the path? Based on all these factors, for a reasonably well-crafted AP where the players have bought-in, it just isn't very likely. On a scenario-to-scenario basis, they players have accepted that they are in an AP, and are looking for quest-givers or bread crumbs, and they don't know enough yet about the overall plot to jump forward, and the level disparity should prevent them from throwing a sudden wrench in the late-game portion of the adventure path. What do you do if they players don't do one of the expected options within a scenario? Again, based on all these factors, for a reasonably well-crafted AP where the players have bought-in, it just isn't very likely. They started at 1st level so they don't have a lot of unusual resources and what they get is governed by the AP, they don't know enough about the outside world to call on unexpected resources, and they know they are in an AP and are looking for a couple straight-forward tactics to handle the goals that have been set out. If they still do something unexpected, you adjudicate the best you can based on the NPC goals and resources, just like any adventure. If they manage to advance to a late scenario out of order, that's fine, level difference will probably force them to back up. If they do manage to kill an important NPC early and unexpectedly, advance someone else from the organization or otherwise adjust the scenario.
