Suppose a dispicable Baron wants to raise taxes. He needs a reason to give the people to lessen the chances of a revolt. He says the taxes are needed to maintain more soldiers. Why do we need more soldiers? Because bandit activity is up and we need to protect the people. The bandits work for the Baron. A simple example of a corrupt noble using bandits to further his agenda.
In my current campaign, I have players who are fledgling warlords in an area. They had a deal with a large group of bandits who were hitting the towns they owned, and they still taxed those towns. Not far off from what you're talking about.
On to answering pemerton myself, I don't use "filler" encounters, but bandits have attacked the party unprovoked. This isn't just for experience points or for some form of pacing, it's to show that the world is a dangerous, corrupt, uncontrolled place (which is very different from the last game world they played in).
Bandits don't have to be "filler", nor do they have to have some deep plot attached to them. They can show your players something about your world. Do they try to kill you on sight? Do they ask for all of your stuff at sword point? Do they only ask for half of your coin and nothing else? These all represent different mindsets in the bandits, and say something about the world. It's informative.
And, since consistency is key for my game (as immersion is one of my top goals), they remain a part of the setting as long as it makes sense for that area of the setting. For example, my players (who are warlords of a region) recently hired a group of adventurers to help them deal with the bandit problem (mainly to get the group of adventurers to stop messing with them, but still). So, the forest area north of their castle is mainly free of bandits for now, but the road to the south will still contain bandits. And, at some point, bandits might venture into that area of the forest again.
Bandits are just like anything else in the world: a part of the setting. I don't see them as "time to fight" (especially since my players have avoided fights with them before) nor do I see them as "filler" (nor do my players). I see bandits (or lack of bandits) as part of the setting, and their presence (or lack of presence) helps shape the setting.
Just my take on it. As always, play what you like
