I think the pacing is the biggest problem with no railroading. I like to have my balance of combat and non-combat: "The party will have a discussion over what to do with the Black Pearl. Then something will attack. Then some role-playing, then another attack. Then some investigation, then the module proper."
When you're not railroading, you're largely (but not entirely) passing the responsibility for pacing over to the players.
I consider this quite liberating, because it means that I don't have to work so hard to provide whatever gaming experience the players are in the mood for that evening: I don't have to try to read their minds. I just let them do whatever they want with the expectation that they'll do what they want to do.
There a couple of potential speed-bumps here:
(1) The players choose to do something that you, as the DM, don't enjoy doing. I haven't had this problem, but it sounds like this is your primary objection: You want to be running some combat and the players aren't obliging.
There's a couple of options here: First, load the sandbox with stuff you like and don't include stuff you don't like. Second, talk to your players about the issue.
(2) The players choose to do something that they don't actually enjoy doing. This seems outrageous, but it's also something that happens frequently with groups who are used to being railroaded. They've been "trained" to see a plot hook and bite it (because otherwise the plot hook will bite them), so when they see the plot hook rich environment of your sandbox they go biting at everything.
If this is a problem, the solution is pretty similar: Talk to them about expectations and the difference in play-style. Make it clear that you don't actually have a preference about what they choose to do -- if they choose to bypass a hook, that's OK.
"Do what you think will be fun shall be the whole of the law" is one of the fundamental commandments of sandboxing.
2) I tried to get them to stay in town by having the townsfolk urge them both to stay and go, but have the mayor offer the fighter sweet armor to stay and be the sheriff. Result, they manage to get on the beach in a boat leaving town before I am able to get the mayor in contact with them, making the railroading obvious instead of a subtle nudge.
Either you're railroading or you're not. Trying to "not railroad" except "when it's really important" is still railroading. Only it's likely to result in awkward and transparent railroading of the type you describe here.
If you're actually interested in giving sandbox gaming a try, then give sandbox gaming a try. Otherwise you're more likely to end up with all the worst elements of both sandboxing and railroading.
3) The party is splitting up over individual motivations -- not fighting, just not the kind of smooth, fun teamwork we get when we just assume motivations and go kick butt. I can't use "you're the sheriff" for adventure hooks because the other two have no interest in being hired on as deputies.
Is this a problem or an opportunity? Let it play out and see what happens. Try providing multiple hooks to the same scenario.
If it's really a problem, then it's time to have another of those chats about expectations. Which can be as simple as: "I don't care what decision you guys make, but you all have to stick together because I can't handle running a split party."
I recommend this essay:
Don't Prep Plots
The author is my DM. He runs an awesome campaign.
On the point of you are not allowed to make plot hooks, I say hogwash. The GM determines what NPCs are doing. And if one of them has a problem, he's free to make it the PCs problem, just as the PCs are free to make problems for the NPCs. Running an environment where ONLY the PCs initiate things and calling it a sandbox is not, it's a wishfulfillment station.
This is utter truth. A sandbox is not devoid of plot hooks. To the contrary: A properly run sandbox is a plot hook
rich environment.
In 3.5, every time you use a monster, include its stat block in a Word file. Print that Word file out and keep it at the table, with its (eventually) hundreds of stat blocks. Where monsters have special abilities that require special info, put that info into the stat block.
This.
One of the things 4th Edition got 100% right was putting every single piece of information you need to run a monster right in the stat block. But there's no reason why 3rd Edition stat blocks can't be designed the same way: I've been running them that way since 1999 (I was in a playtest group).
That's really annoying that you can't run a monster straight out of the book in 3.5.
Not really. It's no more difficult to run a roper in 3.5 than it is to run a roper in AD&D. If anything it's easier to run in 3.5.