The thing is falling is very random. we have instances of a person Vesna Vulovic falling from 2 miles up and surviving. There is also a recent instance of someone falling 90 feet, granted onto snow and walking away.I would put falling damage at 20% of max HP per 10 ft.
a 50ft drop on your head WILL kill you.
if you jump down intentionally, reduce the fall by 10ft and another 10ft with DC15 athletics or acrobatics check, 20ft less with DC30 check.
Personally while the OP can certainly do what he wants in his own game and there is nothing wrong with that its better to tell people up front when you want to change rules. This isn't the OSR where its rulings not rules, its a well defined rules set with most exceptions and structures baked in.
D&D is closer to supers than to fantasy and most attempts to twist anything much above level 6 or so into realism make for an inferior gaming experience.
It can also frustrate players when DM's who can't cope with high power gaming refuse to run high level D&D because they can't relate to it or are afraid of powerful PC's I suspect a lot of people would love to play to 15 or 20 but will never get the chance because DM's lock up and demand whole unnecessary concessions to realism
YMMV but its better for all to accept 5E and other D&D for what they are, power games and if you want verisimilitude or realism in your game, play another game.