I would imagine it's the same reason that in the Dune universe the general "wild" Fremen commoner is far more deadly than the city dwelling Fremen (and I would say even those are more deadly than some soft, water world, commoners). Similarly, from the same universe, there are the Imperial Sardaukar from the prison planet Salusa Secundus.
The Underdark and the Astral places where the Drow and Githyanki live and grow up should have quite some influence on how tough they have to be to survive.
I would guess that when a baby were to be found and raised by common farmers, they won't be quite so tough and resilient when it grows up as the usual Drow or Githyanki.
Sure, the environment and society they live in weeds out the weak. But the weak still must exist at some point to be weeded out, and weeding out doesn't necessitate death. It may
generally be the case just as adventurers
generally come from the "core 4" races. However the weak can be exiled, may run away, and just generally end up somewhere that's not their homeland and not subject to the societal pressures that turn them into violent psychopaths. But even when they do, they're not becoming genetically enhanced, they're gaining class levels in "Violent Psychopath".
I honestly think it'd be more effective to say that to choose to play a Drow or a Gith, you must be at least 8th level(the number doesn't really matter, but run with me here). You don't
lose 8 levels in the way LA would cost you for minor power-ups, instead your character simply has to BE 8th level, to represent the fact that it's highly unusual to see one of these races outside of their homeland at all, much less outside of their homeland and weak. So for all those games starting at 1st through 7th level, you'd have to stick with a race that is less powerful. The Drow or Gith in question would still of course have the base values on-par with a PC race that could start at level 1, so a DM could choose to let someone play a level 1 Drow or Gith, but this is one thing I think Pathfinder truly got right.
LA is not linear. The value of a few small bonuses at low levels is HUGE, but at high levels becomes inconsequential. Which makes calculating LA even more complicated. Which is why the whole darn thing needs to be avoided entirely.
Both Drow culture and environment are quite hostile. I would believe that the only lvl 0 / lvl 1 drow that exist are the house slaves. Similar are the Gith(yanki and zeri [sp?]) societies that are very martial in nature I believe, and live in a hostile environment.
Sure, but what do we frame our lvl1 PCs as? Farm boys who just started to train as their class. Can we say a Drow House slave is really that much of a step up from a human farm boy?