To me, the stat modifiers for a race represent the average variance from human; i.e. a halfling is more dextrous, on average, than a human, but weaker. Eliminating racial penalties just so a player can avoid a stat penalty rubs me very much the wrong way, and I wouldn't be inclined to allow it in my game
It seems to me that you kind of disagree with yourself here.
If the stat scores are meant to represent the average variance then by all means ignoring the stat changes seems way way way more appropriate, because you aren't playing an amalgamation of the average of your race, you're playing a specific individual who might be on all sorts of different places on the bell curve.
Indeed, if they do represent averages then there should be say, Orcs who have 18 Charisma, but they can't because all Orcs get -1 to Charisma, so your player is never going to get 18, but you've just said that there is 18 Charisma Orcs out there so why can't the player be them?
In the end it's really only comes down to balance, you can put an 18 in any stat and be above average for something your race is known at sucking at, the difference between 18, 17, 16 is negligible when comparing an individual to the vast populations of races.
Edit:
For the record I prefer traits to differentiate races and classes. Like, in 2E Dwarves, Gnomes and Halflings got some extra bonuses to save because of their innate hardiness. That's something all of them have, every single Dwarf is different from every single human in that way, some dwarves are more hardy than others, etc.
I think that's way better than adding random stats to try and force the players to conform to the standards of a race rather than the realities of individuals.
Simple stuff, like dark vision/low-light vision, how they sleep, how they eat, makes for much more flavour.
I also agree with many of the comments that a low score can make a character interesting and help promote team work, when everyone has scores above 10 it's not so interesting which is one of my issues with 4d6d1, however it can often lead to dump stats that mean nothing to roleplay and gameplay outside of small specific encounters. Sure the DM can try and separate the party so that the fighter suddenly has to use his 3 Wisdom or something but a lot of the time it's like "Okay here is a door, my wizard backs up and lets Jim the Fighter go forward". It can be fun to mess with them when they end up relying hard on each parties specializations but then that's min maxing and then when they are separated and have to rely on their own abilities you don't want to punish them too much for being interesting. Very difficult DM wise.
But yeah, traits all the way.