I'm discussing two different aspects of the game. One has to do with mechanical balance. Whether or not someone is "big and strong" versus "small and weak" doesn't really fit into that. Based on how I perceive game mechanics are intended to work, the bulky guy and the scrawny wimp have ways to make them essentially equal to some degree.
The other is more conceptual. It is about being able to maintain concentration, or to mentally control and hold onto magic forces despite outside distractions. The scrawny guy's ability to take a punch has nothing to do with their ability to continue a, for lack of a better real-life analogue, mental chant. Sure, you might knock the scrawny guy down. But if he has been training his whole life to hold onto that chant despite pain and distraction, he is more likely to hold onto that mental chant and continue despite being knocked down. On the same token, just because the bulky guy may be able to take a punch better, doesn't mean he would be less easy to distract from continuing a mental chant while being punched. It just means he doesn't get knocked down as easily.
Physical size and strength does not reflect one's ability to focus. But based on the current mechanic, a Fighter 1/ Wizard 4 has a better shot at maintaining their concentration through a spell than a Wizard 20, simply because the Fighter 1 / Wizard 4 took fighter first and has proficiency in Constitution saving throws. The increased mental aptitude and practice of the level 20 Wizard is far more relevant to their ability to maintain focus in the face of distraction (even physical distraction and pain), yet he is mechanically worse than the less practiced individual.