Certainly that only matters on if you care about character power though? Like sure, investing in 'wrong' scores makes your character weaker. But that is only a problem if you see a character being weaker as a problem. Theoretically I wish things were better balanced, but then again, this is not a competitive game, so it is not such a big deal. It actually doesn't matter if some characters are not as powerful as others, as long as the more powerful ones don't completely overshadow the others.
Overall I value verisimilitude over balance, so if it necessitates that goliaths are better barbarians than halflings, then that is fine by me. Granted, ideally there would be agility-based barbarian build that would let halflings to be effective and fierce anklebiters without being bizarrely strong superhobbits.