There is a difference here in possessing athleticism and being a high-level athlete as displayed through on field performance. The knowledge, discipline, processing speed and attention to detail required to perform as an athlete require a significant amount of intellect. This is particularly true for complex team sports with a lot of moving parts like football and basketball.
Will high end athletes tend to possess the level of intellect as high-end physicists? Absolutely not, but Patrick Mahomes, Jason Kelcey and Amon-Ra St. Brown are definitely a standard deviation above the baseline.
Even at the lower end, outside of freak levels of athleticism, someone who is not at least of average intelligence is unlikely to excel in athletics. Savants do exist, but they are outliers. The trope of dumb athlete is just not very reflective of any locker room I have ever been a part of.