I have similar concerns to you. As a player since early 1e era, I had 1HD at 1st level as a sacred cow all the way until 4e era.
While I never adopted 4e and didn't like 3HD at 1st level solution to the same problem you are trying to suggest, simply challenging the sacred cow forced me to do some soul searching about the game.
What I ended up implementing as my solution was to apply bonus hit points to all characters based on size class. My table looks like
Fine: 0
Diminutive: 1
Tiny: 2
Small: 4
Medium: 8
Large: 16
Huge: 32
Gargantuan: 64
Colossal: 128
We can discuss the exact effects of that at length of you want, but after a half dozen years of play testing I'm very happy with the change. Overall, it works very well.
Two problems to call out:
a) While you pick up verisimilitude in some areas (notably the "house cat problem" and the "large herbivore problem"), you will have some off setting losses of realism in at least one other area. Notably, while you now know how the farmer bullies the house cat, you can no longer explain how he hunts deer with a bow. Since perfect realism isn't the goal, this probably isn't an issue.
b) You'll probably need to address balance in a 5e context. This was fairly easy in a 3e context because being 'small' was almost strictly a benefit as long as you were sheltered from problems with movement rate - the above actually adds balance to +0 LA races in 3e IMO. But I haven't tried the rules change in 5e and you'll need to evaluate whether losing 4hp is a big problem for small sized races.