I tend to base it on several factors: the state of the combat, the intelligence of the opponent(s), the hunger level of the opponent(s), and whether killing a PC as opposed to just knocking it out holds any material benefits for the opponents. (an example of the latter might be an intelligent undead that's downed a PC and knows that finishing off said PC will generate another undead under its control and tilt the battle odds in its favour a bit)
Lan-"and if unsure I just roll randomly"-efan
Adventures in Middle Earth has an ability that some monsters have that makes a character fail their first death saving if their attack reduces them to 0 hit points. I've adopted it for a number of creatures. It's a nice middle ground.
As for whether a monster will continue, some in the past (like the Peryton) specified that they will continue to attack and ignore other threats. I think that's pretty rare. To me:
Animals, or animal-like monsters (predators) will, if anything, attempt to take the downed creature to a safe place to eat. Predators aren't generally interested in attacking to kill everybody there, just one to get a meal.
But they very well may have spent one or more extra rounds killing the prey, because to them it's attack-attack-attack until the creature is no longer trying to escape and/or defend itself. But they only way they know if it's really not moving is to stop the attack, and if they stop it too soon, it might escape. So they probably overshoot pretty frequently.
Many predators are used to fighting for their food too, but a solid hit against a predator will likely cause it to attack the source of the attack, or retreat. Most animals in my campaign have a fear of fire, too, so a torch is a very effective defensive weapon.
Most of the time, though, creatures like those are driven off by the fact that the party is fighting back in my campaign. It's tough enough for a predator to get enough food to survive, and survival depends on being healthy. No sense in sticking around when the prey fights back that well.
Mindless creatures, at least those oblivious to the risk of death, might very well continue to attack. A ghoul, for example, might not only immediately start to feast, but might have to fight off other ghouls. Like predators, they may try to pull the downed creature away from others. Zombies will probably continue to attack the creature until they are attacked by another creature.
Moderately intelligent evil creatures, such as orcs, etc., will gladly take an extra stab or two at a corpse, provided they aren't in immediate danger themselves. If somebody else is attacking them then they'll continue the attack instead.
Most venomous creatures (like snakes) will bite then retreat, waiting for the venom to take effect. The black mamba, however, is an exception to this rule, and will strike multiple times and pursue their prey, for example.
So I guess in response to the OP, I use a considered approach. Most of these types of tendencies are addressed well ahead of time, and the adjudication portion comes with unique circumstances. In a recent scenario, I had a predator with young, and determined ahead of time that they would defend the young aggressively, well beyond when they would disengage a hunt. But at about 25% hit points, they would disengage and leave the young to whatever is attacking them. This behavior is found in many animals in the wild. Just because a creature might be a "monster" rather than a normal animal, ecology works roughly the same way. They still have to survive within the world. And it's not a consideration of how they survive or how they act when confronted by an adventurer. Most individuals of a given species may never, ever run into an adventurer. Instead, they have to deal with their normal prey or sources of food, and their normal predators.
I do dig a little deeper, such as figuring out how their special abilities work within the wild. And this consideration is within the world, not the rules. For example, a cockatrice. What would be the purpose of their ability to turn creatures to stone? Even if it was spontaneously gained due to a magical event, once they have the ability, how would it change their behavior?
One option is like the basilisk, which is noted as being able to eat the creature that is petrified. In which case their ability is very similar to snake venom - immobilize the prey, so they can feed. Which means that the idea of statues left after a basilisk attack is unlikely - they use their ability to gain a meal. Being seemingly reptilian in nature, one meal probably lasts a long time. So if a group of creatures were petrified, their might be some left. But I would guess they could eat two or three at a time.
That doesn't seem appropriate, for the cockatrice, though, and the 5e stats indicate that the petrification lasts 24 hours, and they are omnivores eating small creatures in addition to berries, etc. So for me, it's defensive. If they are threatened, they attack, turn their predators to stone, and escape. So the recent encounter with a flock of cockatrices was essentially a flock of chickens that exploded out of the underbrush, made some attacks on the way, and scattered. They were a nuisance, other than the fact that I retain a system shock check when turning back to flesh.
I'll also modify a creature to account for its abilities for the same reason. For example, what benefit would the displacement be for a displacer beast? Sure, it has a defensive component, but if that was a natural ability that it has had for hundreds if not thousands of years, I think that as a predator it's most likely to be used in hunting. My answer? Their tactics (supported by a new mechanical ability).
Displacer beasts are pack hunters, and they attack by scaring the prey in an attempt to escape, which mechanically means it triggers an opportunity attack. But their displacement means they aren't where the prey thinks they are. So they have advantage on opportunity attacks. And that's their hunting tactic - drive the prey to attempt to escape and into a trap. More often than not, it's not making a primary attack, only opportunity attacks, then pursuing and stalking the prey if it escapes.
Opportunist: The displacer beast has advantage on opportunity attacks, and they make two attacks, one with each tentacle, when making an opportunity attack.
Add in the fact that it's similar in many respects to a large cat and it has six legs, I've also added a climbing ability. It's really a terrifying monster as a result.
As for whether a creature would attack an adventuring party? It depends.
A bear might attack if surprised, but will most likely run away. A dire or grizzly bear, on the other hand, might be noted as being more aggressive and press an attack for a couple of rounds before running away.
This is likely the case with any predator that doesn't view humanoids as prey. A tiger or lion might attack and kill a human, but unless they aren't able to get their regular food, won't press the attack. On the other hand, in regions where their natural food sources are in decline, then you find the man-eating tigers that have learned that humans
are edible. Furthermore, they often find they are easier prey to catch than many of their natural foods.
--
Would adventurers cast
silence on bats? Well, if they were trying to be quiet to not disturb the bats, perhaps. Sometimes it's easier to cast the spell on the bats, then ensure that you all stay within the effect of the spell.
--
In the end, for me anyway, the more you start with reality as a starting point, the more resources you have to answer these questions before they come up at the table. You can alter them to suit your needs, of course. "Rulings not Rules" doesn't mean you won't have a solid framework to work from.
