Some thoughts:
Retreat is a result of two situations:
1: Poor party planning.
2: Poor DM planning.
There can be a lot of reasons for poor party planning: Maybe they didn't pay attention, maybe they're not high enough level, maybe they're just kinda dumb, maybe they utilized their class features and tactics poorly. But poor party planning is most often the quickest route to failure.
If the party was at the appropriate level for the challenge, prepared well enough, played well enough then the possible alternative is the DM may have messed up in their encounter design, as I think you OP, encountered in your game. Poor DM planning is thankfully adjustable on the fly, perhaps the sudden influx of new monsters results in a decrease in their tactical capabilities, or not all of the new monsters are on the same side or other reasons as necessary.
In my experience retreat comes in two forms:
1: Sensible party decisions.
2: DM intervention.
-Capture instead of kill by the opponents is a form of DM intervention.
In my experience, parties are rarely sensible. Even the smartest players and smartest characters can be overcome by a few bad decisions or the sight of treasure. And of course there's the age of axiom of: no plan survives contact with the enemy. So generally speaking parties will choose the option with the highest potential for reward, which is often the option with the lowest chance of success. Since the party usually chooses to take the high-risk-high-reward option first, this means that when retreat becomes the only viable solution to the situation, the party has risked all and lost. This tends not to set well with parties, so it doesn't happen.
DM intervention is a touchy subject. Some people are "victory or death!" sorts and they tend not to appreciate the DM stepping in, in whatever manner and saying "Sorry guys I'm going to take victory off the table for you because really it ain't gonna happen." Well, who is the DM to make that judgement call? Sometimes the party survives the worst of situations surprisingly better than you'd expect so it feels unfair for the DM to jump in and call the fight. Some folks are OK with the DM jumping in and saving the party's ass wen a fight goes sour, so long as it's pretty obvious that victory is such a far-fetched possibility that there's really no alternative.
Killing downed party members:
This really comes down to the sort of monsters you're running. I tend not to kill downed party members when I have the chance, and there's a variety of reasons for this: Perhaps the beasts you're fighting like live food. Perhaps the villains want some slaves or minions or to interrogate you. Perhaps I'm just a softie.
I think there are very few monsters that would kill you when you're down. Strategically its the best move but the problem is that very few monsters are that highly strategic. Unless the monster has specific orders to make sure you're dead, once you're down most of them are going to assume you'll stay down and move on to killing the rest of the party. Highly strategic monsters should not be played as genre-savvy. Genre-savvy villains suck. They're boring to run and they're even worse to fight because they'll always always go for the kill. Highly strategic monsters are likely going to want to use the party for strange rituals, for fun and for torture and potentially interrogate them, giving the party ample opportunity to escape thanks to stupid henchmen and faulty prisons.
----
In the game I've been playing there's been a lot of retreat. We're not a very strategic party, several players blatantly don't know their class and we're easily swayed by the lure of big treasure. The DM, however is fairly generous with us and gives us ample opportunity to retreat. We don't always take it, and tonight he had to force us to retreat ala: powerful NPC intervention. Which was fine in several ways, the DM got to point us in the direction we should be headed in order to level and gear up and raise our reputation with the world, etc... It also answered some unanswered story questions and generally the party was okay with not being dead.
The DM almost never kills downed players unless the enemy has a specific reason to want to be sure they're dead.
It's important to remember that in 5E, being "downed" or "at 0" doesn't mean you're easy to kill. You maintain your AC, but lose your dex bonus, and by the rules you either have to: deal damage equal to that person's maximum hit points (-100% health) OR deal damage 3 times (each time you take damage counts as a failed death save). There's no auto-kill rules for someone to walk over, slice your throat open and boom, you're dead. While there's nothing stopping this from happening, it's worth keeping in mind that doing so often comes off less as "You're severely wounded, this guy wants to make sure you don't get back up." as the DM saying "I don't want you to live."
First of all, I whole-heartedly disagree with the idea that retreat must be the fault of either poor party planning or poor DM planning. Retreat need not be the result of player or DM failure. Furthermore, as a DM, I make no pretense whatsoever that the "encounters" in my campaign are level appropriate, and I usually don't plan them strictly as encounters at all. I use some discretion and math in determining how many hostiles of which types to put in a location, but once all the PCs and NPCs are hanging out, things just kind of run their course.
Generally speaking, I am a far worse tactician than all but your most unintelligent of monsters, so the sudden influx of creatures does decrease my tactical capabilities (as I'm now more likely to forget about special combat abilities from at least one of the mobs). Beyond that, I don't go out of my way to adjust-down a fight just because it's over the heads of the party. I might go out of my way to tell the party just how dangerous their opponents are.
I've forced a retreat once and, honestly, it was a terrible session and I wish I hadn't done it. I don't think, as the DM, that I should force the players to keep their characters alive. However, once the players decide to get out of Dodge, I'm happy to work with them on it. Retreat opens up so many more options for creative problem solving than your standard combat.
I also disagree with you that attacking downed PCs is a genre-savvy or meta-gaming action on the part of monsters. A PC who has gone to 0 hp is almost like a boxer who has been knocked down. They're briefly out of it, but also clearly still alive and breathing. A monster would see this, and understand instinctively, "this guy's still a potential threat." Even without the possibility of magical healing. (Which monsters are also well aware of, since they live in the same world of magical healing that the player characters do.) Genre-savvy is understanding the unique motivations of PCs as controlled by detached otherworldly beings stuffing pizza and beer down their faces and taking several minutes to plan out every 6-second action. Basic awareness is understanding the general mechanical underpinnings of the world you live in.
I think there are plenty of reasons why a monster
wouldn't attack a downed PC, but I think many of us, my earlier self included, tend to rationalize away the reasons why a normal opponent
would attack a downed opponent. I'm re-evaluating those reasons. Will the monsters stand over the unconscious PC hacking and slashing until that PC fails their final death save? In the middle of combat? Probably not. But will they use their second attack as part of multi-attack to give another hack at that PC as he falls to the ground? Absolutely. Happens in medieval war movies all the time. The character might still survive that follow-up hack, but it makes the stakes a lot higher.
What I like about what I'm describing is that it's not a house rule. It uses the death save mechanic exactly as written. Even the unconscious PC still has some defenses against monster attacks, and that's as it should be. Falling unconscious in combat should be
really dangerous. Characters should be able to die in a combat that isn't a total party kill. This will never happen in 5e once you get past 2nd level if the DM applies some sort of Red Cross/Geneva Convention to monsters.
What's the point of a "victory or death!" approach if there is actually no risk of death?