Damage is irrelevant when craftsmen in your fledgling barony complain your tax collector charged them double, and your investigation yields that he skimmed a few coins in the past, gave breaks to a few poor folk who had a bad harvest before winter, and made up the difference with what he perceived as the richest people in town. The tax collector thought he was doing right by the people, which is what he emulated your party's leadership doing, but if you fail to act, you won't have anyone to build your Temple (which you promised you would to the Church since they loaned you a significant amount of resources to found your town).
I digress. This is a thread about combat, and in D&D, a good offense is the best defense. It's been that way since Day 1, but really, it shouldn't have one iota of impact on what you play.