TwoSix
Everyone's literal second-favorite poster
It's complicated.These are the very words I was seeking before my morning coffee truly settled in.
Like you say, there are narrative truths to each species that, I think, are more important to most players than the mechanical benefit. Being an elf means something, even if you don't get a +2 to Dex and proficiency in longswords.
The mechanical benefits help, of course, but I think one of the reasons a lot of discussions on species mechanics gets lost in the weeds is because for most players mechanics are a secondary incentive. The narrative truth is the primary incentive.
The more potent a race's mechanical benefits are, the more likely they are to be the driving factor in character creation. When they are less potent (or nonexistent), then aesthetics and concept will become the driver.
But, a lot of people don't want their racial/species/ancestry choice to be only as mechanically consequential as the color of their armor. Since it's a significant portion of the character's total concept, they want it to have mechanical weight commiserate with its importance.
Also, since race/ancestry does carry a lot of presumptions and narrative weight, trying to determine what the race can actually do narratively can be fraught within the game, with a lot of DM adjudication and potential argument. Just because you might think your halfling can gain cover behind an ally, doesn't necessarily mean the DM will agree!