The way the game used to be played in older editions, XP was awarded for killing monsters- I think that's where it comes from. Nowadays, I see a lot more of the philosophy that "a creature generally won't fight to the death if it can somehow escape," and playing monsters 'smart' means more of them will flee, perhaps even abandoning a lair for good, in order to preserve their lives. In these sorts of situations, I'll award full XP for the 'defeat' of the monster as opposed to slaying it. When I was a teenager playing with friends, failure to slay meant "no XP for you, biotches!"There's something baked into D&D that makes monsters surrendering or running away just not as satisfying as killing them. My players will often go out of their way to hunt down a retreating foe, even if it's just one goblin or even a single crab (this happened!).
Hunting down a retreating foe is a player choice. Some people are just 'finishers' and don't like the idea of leaving a live enemy at their backs, especially if there's a chance of encountering it again. When I was a kid, if I was playing Samwise Gamgee I would have chased Shelob right back into her cave and tried to finish the job. Fortunately I've grown out of that sort of single-mindedness playing DnD!
