Sadly true, but this unfortunately is the failing of skills in general in 5E, and especially Athletics.
Very few people who would have proficiency in Athletics would be equally adept at climbing, lifting, jumping, running, sprinting, swimming, etc.--everything this cluster-skill does, which let us not forget also is involved in combat with grappling (wrestling) and shoving. Why have it be so many different aspects of physical ability? Couldn't you just give martial classes two more skill proficiencies and let them pick their specialty? After all, wouldn't Strength be the better indicator of "general" ability in Athletics? You have 4-5 skills for the mental abilities, so why not have 4-5 for Strength: Athletics (climb/jump/swim), Brawn (lift/shove?), Grapple, Run (jog/run/sprint) or something??
After all, they didn't group deception, intimidation, and persuasion together? Yet all of those skill are really just a single skill: Influence if you look at it like Athletics, with a different method of how you influence someone-- lie to them, threaten them, or convince them.
This emphasizes the above point. You might be fast, but I was faster. Yet you had better endurance (despite my efforts at cross-country in the fall season) so would beat me in the 800 (maybe even the 400? My best was only 52.5, so decent but not great...).
In short, I was a better sprinter in all likelihood, but you were a better runner. I also did long jump and had a best of 20.5 feet in high school, but I KNOW I didn't have a STR 20!!! Heck, back then I could barely do a single pull-up, but I could leg-press over 1,000 lbs. yet I weight just 145!
In general, for "Athletics" I was a decent climber, poor lifter (upper body) but great lifter (lower body), good jumper (my best high jump was 5'9" and I am just under 5'8" tall), good jogger (sub-5-min mile) and runner (52.5 400-m), great sprinter (10.9 100-m), and a decent swimmer, not to mention decent grappler/shover maybe...
A trait or feat for speed alone could be the good way to go, after all we have the Powerful Build trait for lifting, but I would rather see the rules/mechanics and skills be more robust to reach the goal because I don't want traits for every aspect of "Athletics".
It is one reason why I liked the skill web in Shadowrun (2e and 3e) with general skill/ specialization/ and concentrations.