Defeat in combat doesn't have to mean death. I know that in D&D that's the default; but it's possible to narrate being dropped to zero hp in other terms. In 4e D&D, this is a possibility regardless of attack mode - instead the target is knocked unconscious. The one time I "TPKed" the PCs when GMing 4e, only one of the PCs literally died (the events of the combat dropped his hp below zero); the others I narrated as unconscious, being taken prisoner by the Goblins whose spirit summoner had conjured the undead spirits that defeated the PCs.
4e even made that PC death easy to handle. The Goblins took his body, too, back to their caves, where he was laid out as an offering. But the PC's god brought him back to life: as GM, first I free-formed the conversation between dead PC spirit and god, and then - when it was determined that the god was indeed sending the PC's spirit back to the mortal realm - I deducted the cost of a Raise Dead ritual from the treasure parcels for that level.
A non-D&D RPG that I like a lot is Torchbearer 2e. It has a conflict resolution system that is very different from D&D's, and is applicable to a wide variety of conflicts (eg various sorts of social conflict; dealing with spirits; flight and pursuit; as well as combat). Within the scope of combat, it distinguishes Kill, Capture and Drive-Off conflicts - the mechanical resolution structure is the same, but the relevant stats differ, and so do the stakes and potential consequences. Only a Kill conflict puts
death on the line as a possible consequence, and a Kill conflict will come about either because the players choose to initiate it (
here's an actual play example with two Kill conflicts in close succession), or because the players failed a check, which thus licenses the GM to introduce a twist, and the GM's twist is a Kill conflict (
here's another actual play example, with two such instances in close succession). In more than twenty sessions of play, we've had pretty few Kill conflicts - maybe half-a-dozen or so - but the combats still have plenty of heft and consequence. The PCs have found themselves
driven off, or
captured.
As well as combat, death is also a possible consequence if a player fails a check for their PC when their PC is Sick or Injured (these are the two most serious conditions in the game). But the GM has to announce, before the check is resolved, that death is on the line, so that the player has the chance to put as much "oomph" into the check as they want to. (
Here's an example of this in play.)
And if a PC does die, the player may (if they have the requisite Persona point unspent) declare that their PC has The Will to Live. This is a mechanical process, where the PC undergoes some non-trivial mechanical adjustments, but miraculously survives what otherwise would certainly have been fatal. (The first play report I linked to in this post has two examples of PCs having The Will to Live.)
The mechanical details of TB2e, and the resulting scope of stakes and consequences, aren't directly portable into D&D. But the 4e example I've given shows how something in the same general neighbourhood is possible.