I think intelligent monsters are actually the other way around. Unless they're zealots of chaos or completely feral, they are reluctant to kill because they want to show enough strength to drive off the invaders, but not give the invaders an incentive for revenge or back them into a corner where mutually assured destruction is the only course.
I think a "no quarter given or asked for" situation would be uncommon, since everyone wants to live and theres always the chance you'll be on the losing side...
But that strategy doesn't really work against adventurers. Adventurers don't say, "oh no, they knocked out one of our guys. we will run away." They say, "oh no, they knocked out one of our guys. We need to get our guy back." Sure, if you kill one of the PCs, they will be back for revenge, but if you don't kill one of the PCs, they don't need to come back. They're already in your monstrous home, invading it and stealing all of your stuff. As an intelligent monster, I would assume that a putative group of adventurers would probably run away after seeing one of their own get killed, then surveying the battle and saying, "at this rate, another 2 or 3 of us are also likely to get killed." On the other hand, very few adventurers will run away after surveying a battle and saying, "3 or 4 of us might get knocked unconscious, but at least one or should survive long enough to win the day and then revive our companions."
Now, sure, if there is the possibility for some sort of prisoner exchange, some groups will attempt that. Unfortunately, a prisoner exchange depends on strong communication between parties and a certain degree of trust.
Being knocked unconscious, in of itself, just isn't that scary. Sure, there's something like a 3% chance that you'll crit-fail one death save and then fail another one, killing you in 2 rounds. But, on average, an unconscious character has 4 or 5 rounds of death saves, and a 55% chance that they will stabilize without any intervention whatsoever. There's even a 5% chance that they'll pull through on their own and be back up on their feet, swinging that greatsword. When you're a monster, repelling invaders, and that's the most you've got to scare your invaders with, you're not going to have much success.