As I just pointed out in another thread, the big issue here is Enlarge and Reduce. Both spells alter your size with built-in effects to simulate the change. You don't want to make a large race that just does an extra d4 damage on all weapon attacks just because, for example. Especially since, due to 5e's emphasis on simpler rules, you can cast Enlarge on a Large creature with no problems, so if being Large had an equal benefit, you'd be able to make someone Huge for +2d4 damage.
Reduce has the same problem, if you wanted, say, a Tiny race. Subtracting 1d4 from weapon damage will just mean that people will mostly play Tiny casters, or classes that don't care much about their weapon die, like Rogues.
Which, I mean, is logical, but I hate pigeonholing races in this fashion, to where one race becomes "the best" or "the worst" choice due to some racial ability.
"No, don't play X race, play Y race because they get +d4 damage on all attacks"...yeah no thank you. Not unless there's a significant downside. Which being big might be, but it would also be a headache for a DM.
It's like why dedicated characters that are meant to ride horses are such a problem- dungeons don't really work with Large mounts, so you quickly have people playing Gnomes and Halflings so they can bring their loyal steeds with them.
Ditto for classes that are supposed to have Large sized pets.