Your PCs must be the best-behaved party in all of D&D in that case, as they have clearly never:
- attacked an innkeep after being shortchanged
- captured a street urchin picking pockets
- started (or tried to stop) a town riot
- been involved in (or, as happened in my game, been the quarry of) a torch-and-pitchfork brigade
- participated in a bar brawl with the locals
- needed to neutralize a town guard in order to sneak in
- needed to neutralize some prison guards in order to reach someone and free him
- been involved on either side as a local farm gets raided
- etc.
All of these require at least some mechanical representation of the commoners involved, even if their only further contribution is to flee, die, or do nothing.
Lanefan