I tend to compartmentalize the concept like this:
A character of mine may "lose" something, like a magic item, a level, or it's "life".
As a player, these in-game setbacks do feel like losing (and depending on the circumstances, may even provoke my ire). So I agree with you on how things may...