The thesis that the key problem with retreating is not thinking about retreat correctly, and the unrelated conclusion that the monsters have good motivation to retreat early.
I am not really sure. When the monster initiated the fight then yes, because it presumably had a purpose (as you mentioned: hunger, greed, sport or something more complicated if intelligent): even in these cases a pursue wouldn't necessary be unlimited, it could very likely stop once it kills one for food or once it acquires what he's greed about, ignoring the rest.
Well sure, but you are arguing into my point rather than out of it. One of the PC's getting eaten so that the others might escape is not a tactically or socially desirable outcome. It is in fact the logical result of choosing to retreat, which is one of the reasons players don't choose to retreat very often. In my experience, really skilled parties would rather risk TPK than retreat in a situation where one or more of the PCs aren't going to make it. That's because the vast majority of deaths in D&D occur when party cohesion breaks down and the party stops acting as a unit.
The vast majority of PC deaths in my current campaign have occurred as a result of an ill considered decision to retreat and the party getting pulled down in the rout phase. Yes, the ghouls will stop chasing you when they have carcasses to feast on, or the hellhounds can be distracted by pulling down the slower members of the party, but this is not a tactically sound solution. (The players are getting better. They are learning that if you think you might want to retreat you have to prepare for that ahead of time and not expect that you can retreat just by running away.)
When the monster is attacked by the PCs, once the threat is no more its main purpose (survival) is fulfilled, it won't necessarily pursue even if winning. If it has something else to defend other than itself, I actually think it would not pursue at all (would a group of guards leave their post to pursue?).
Put it into practice and see how it works. It's rare in my games that monsters that the PC's are in combat with have the main purpose survival unless the PC's have invaded the monsters lair and the PCs have the intention of fighting to the death.
Animals generally stop being a problem for a PC party somewhere around 2nd or 3rd level. PC's are rarely in combat with anything that has 'flight' more as its response to danger than 'fight', so combats with antelope are for example pretty rare. Watch some nature videos though, of say lions fighting cape buffalo or lions fighting warthogs or lions fighting elephants. Even in the case of a warthog, while pursuit isn't unlimited, if the warthog gets the upper hand it won't immediately break off pursuit. It's motivation here is survival, but part of that survival mindset is teaching a lesson to the predator not to try that ever again.
But this is vanishingly rare. D&D combats are usually over quickly. Even with a "dumb" animal, it's not like there is a long gap in D&D between, "Oh I got hurt." and "Oh I'm dead." And even if you try to exploit that gap, in the round or two it takes to retreat it's dead from missile fire. PC's very much will pursue anything to dead for a variety of reasons both in game and out of game (they fear being cheated out of XP for example). It's very rare for any monster I decide will rationally retreat at this point to actually get away. More usually, if its something like orcs or velociraptors that attack in a group, a couple might survive the decision to pick a fight with the PCs for the same reason some of the PC's might get away if the whole group routs.
It's not that attempts to retreat don't occur it's that they don't work and the intention to retreat doesn't stop injuries from being severe. I mean, it's not that unusual for a retreat to include some PC's unconscious body being slung over some larger PC's shoulder (or mount).
Heck yes guards will pursue. Not only will they pursue, they'll raise a hue and cry to organize a larger pursuit. You try in the real world assaulting a police officer or some such and see how tenaciously they'll pursue you. Rationally there is a cost to "letting them get away" that is the same reason the warthog pursues the young lioness that thought it easy prey, or the mother elephant keeps charging an intruder that got too close to its young even though the intruder is already retreating. You have a motivation to survive but part of that motivation is making sure nothing jumps you in the future. You at the very least want to put the fear of you in the thing, and vengeance is not a purely human motivation. I'm sure there are exceptions, and oath bound outsiders sworn to protect a lair would be an example of something that literally can't leave to pursue, but those are exceptions.
And speaking of which, generally my players tend to use the logic, "If we could retreat successfully, we could probably also
kite the monster successfully."
The tiger example I brought up, was actually in my mind meant as a random encounter, so that the PCs don't have any special motivation of their own for truly killing it. I think that most typically a DM would just have the tiger (or whatever) fight to the death. But wouldn't it make sense that the tiger attacked the PCs for food? And then after being wounded (not much problem here narrating the HP loss of a random encounter monster as real wounds) the predator would probably realize its mistake and run for its life.
Sure. But a "tiger" is a rare encounter that isn't really relevant after about 3rd level. And if I wanted to justify a tiger fighting to the death, I'd just have the tiger be a mother tiger with cubs hidden nearby that she refused to abandon. That would be a far more interesting encounter than "A tiger attacks a large armed party but ooops it got in over its head." My advice would be to just not have encounters like that since they don't really fufill any particular aesthetic of play that the players might have - it's not challenge, it feels forced and unrealistic, and it offers no narrative scope. But mommy tiger defending her cubs is a challenge, feels justified, and offers narrative scope, "Look tiger kittens...Oh, she was just defending her cubs.... Aren't they cute." and even RP opportunity assuming you can speak with tigers, "Oh.... she just thinks we are a danger to her cubs!" "I want to eat you" doesn't leave much in the way of RP opportunity anyway.
Nor do I see what this point has to do with the topic that started your line of thought, which is can we use retreat as a means of justifying the PC's being healed fully the next day. Are we trying to justify the tiger as being fully healed the next day?
Basically I'm saying I had these thoughts 30 years ago, and after 30 years I no longer consider 'retreat' a functional solution to much anything for either PCs or NPCs (except turning planned combat encounters into planned chase encounters or visa versa). Aside from the fact that a vast swath of monsters don't retreat or can't as a matter of practice retreat, it's very gimmicky, hard to pull off, and doesn't add much to the game. It's best to just avoid having encounters where the monster has only a small motivation to enter combat, unless that small motivation is used as a justification for turning a boring combat encounter into something more interesting.