Wow. My players always flatly refuse to pay the bandit toll. They will always attack bandits. No matter what level they are. LOL.
(One day I'll work up some high-level bandits who can kick the low-level PCs' butts and make them pay the toll until the PCs finally get high enough level themselves to be able to challenge the bandits and win.)
That's always fun.
In my 3e/3.5 campaign, the group (1st level) was passing through a road where the was that exact setup, a higher level bandit (4th-5th or so, with his big group of outlaws) demanded their gold. The group, predictably, refused. The bandit and his gang beat them senseless and took ALL their stuff, not just the gold. The group slunk off.
Much later. The same group, now 13-14th level, passes through the same area.. They get waylaid by the same highway man and his crew. But the highway man is basically the same level, and they're, well, not. 1 or 2 rounds later the entire gang is down, and the group take all of THEIR (the gangs) stuff. Not that they had any use for it, pure vengeance. The group had as much/more fun with that encounter as they had with any of the nail biting actually important ones. They got to experience the fun of being better than they used to be, right up close
When the group needed a tax collector for their keep, they tracked the highway man down and gave him and his gang a job.