It is impossible to prove that a being cannot be redeemed. Violence is only justified when there is imminent danger. The question is how many evil acts and how severe are they that would justify simple executing the being to prevent future acts?
How does the average PC gain enough knowledge of Orc Henchman #3's backstory, so as to decide if stabbing them and picking up the adventure's allotted treasure parcel is okay?
Unless anyone wants to come back to it, I'm going to skip the
treasure aspect of this, as I think that raises a whole lot of other questions around property rights, robbery, etc.
But I think that limiting permissible violence to imminent danger - which is a pretty standard contemporary approach to the morality of legitimate violence - will hinder a lot of FRPG play. That's why, upthread, I suggested two other bases: consensual violence, and retributive violence.
If Orc Henchman #3 is part of a gang of bandits, that may be sufficient to justify the use of violence against him. If it turns out that these bandits are actually not bad guys (eg they're Orcish Robin Hoods, or a legitimate Orcish national liberation movement) then even though retributive violence wasn't justified, it may be excusable in the sense that the PC wasn't culpable in thinking the Orcs were baddies prior to getting that additional, exonerating information.
If Orc Henchman #3 is part of a tower guard, like the Orcs of Cirith Ungol in LotR, then I think this is where consent becomes relevant. By fighting in defence of their tower, rather than surrendering, they are rendering themselves permissible targets of violence. In real life it takes more than the preceding sentence to explain why it's permissible to attack soldiers, and in any event many people doubt that someone can consent to being killed in this sort of way (once reason why consensual duelling is illegal in many, probably most, places). But I think in the context of FRPGing the fact that the Orc is a soldier who chooses to oppose other soldiers (ie the PCs) is enough to do the moral work.
Now if we are talking about a situation which looks like nothing more than a home invasion by the PCs - the Orcs aren't bandits and so aren't liable to retributive violence, and are fighting in defence of their homes and their fellows - the case for justifiable violence seems harder to make out. Maybe the Caves of Chaos gets close to this?
So I get the deal with orcs and such, but how do people feel about evil non-humanoid creatures?
I'm not totally opposed to sentient creatures that act in mostly cruel ways. I just think it should be done with a critical look at the tropes and they should not be inherently cruel.
For me, I feel it's about the tropes and themes that are being drawn on and re-articulated. And it's of the nature of fiction that this can happen in all sorts of ways.
For instance, fighting alien eaters-of-the-brains-of-the-living, or alien slavers - using
alien in the sci-fi sense - seems innocuous enough: those seem like cruel practices, and the violence used against the aliens seems like legitimate defensive and/or retributive violence (given the broader conceits of FRPGing).
But I think a game that focused on a military campaign to
wipe out Mind Flayers or Neogi might be distasteful (at least) because of the way it implies the permissibility of genocidal violence, and of widespread ecological violence as well perhaps.
Again speaking just for myself, this is yet another reason why I prefer FRPGing to be focused on
the situation - the call to action that confronts the protagonists - rather than on
the world. Let me fight
this demon without worrying so much about the moral meaning of wishing the end of all demons everywhere; or if I'm playing a non-heroic PC, let me focus on
this moment of criminality without worrying about the moral meaning of criminality as such.