As others have said, it's not something that can be nailed down simply; that would defeat its purpose. It's a composite of common knowledge, community standards of conduct, the limits of what we expect a typical adult human being to be able to think or do (e.g. we don't expect a typical human adult to be able to see through walls or factor a seventeen-digit number in less than one second, but we do expect them to avoid unnecessary risk and consider alternative courses of action), and other relevant factors. As with many "tests" in legal proceedings, it inherently cannot be pinned down to an explicit formula, because that would create both exploits and overreach.I don’t have a dog in this fight but one of the best games I’ve DM’d was Way of the Wicked by Fire Mountain Games.
<snip bad characters doing bad things>
Who decides what a reasonable person thinks is acceptable?
As for Way of the Wicked:
Were your players on board for the stated premise and accepting of the concept as delivered? It sounds like yes, which means they were engaged.
Were your players there by consent, and able to withdraw or address concerns with reasonable speed as they arose? It sounds like yes, which means they were willing.
Were you and your players treating one another with respect and not deriving joy from causing harm to one another, or other peripheral participants? It sounds like yes, which means they were being (within reason) positive.
Your group roleplayed horrible people doing horrible things. That is, and should be, pretty clearly distinct from using speech or performing actions that apply to real, living people, whether present or not. Such gaming naturally isn't for everyone, in exactly the same way that games with a highly sexual or graphically-violent tone are not for everyone. A reasonable person would be presumed to either be on board (and thus the game is for them), or be able to depart once they realized they weren't on board--and failure to let them do so would be demonstration of at least one of the above things failing to be upheld (positivity, willing consent, sincere engagement).