And so now they're collecting brigands...

[VirgilCaine]

I like the "mistaken for slavers" bit. If they want to collect brigands easily, they're going to have to deal with the downsides.

 

[LordVyreth]

And, of course, there's the inevitable scenario when they capture a bandit that manages to escape, and takes all the prisoners collected up to this point with them. Or the friends of the bandits, or just another group entirely, frees the bandits while the party is busy during a dungeon crawl. And with the party's habit of taking the stripped weapons of the bandits with them, the escapees will be re-equipped instantly.

There are also plenty of adventure hooks that could help the party. What if they were about to be attacked by a far greater force, particularly one that isn't known for its mercy, and the party deputizes the bandits in exchange for pardons? What about using them for dungeon scouts in exchange for the same deal? What if the cleric or paladin manages to win a few converts before going back to town?