You're making a different point. I didn't say that it didn't make sense that physical skills couldn't play a part in social or intellectual challenges. Mechanically, they get boned since they are only useful for 1 out of 3 types of challenges.
This is why RAW is superior here, because sure most of the time it does not make sense, but perhaps you have a case where bargaining for the release of prisoners involves running across down cutting deals before the discussion finishes (Athletics) or is stretching on for hours and hours and its a struggle to keep pushing (Endurance) or maybe you are playing a character like Kronk's shoulder devil, or a variety of other creative options.
And that's the point I was making: RAW already recognizes that it doesn't make sense to have, say acrobatics in a prisoner release bargain, that's why the DM selects the primary skills and doesn't pick Acrobatics as one of the skills. It also recognizes that Diplomacy doesn't make sense tracking a dragon through the forest. However, RAW also allows for a creative player who comes up with a reason why Acrobatics helps the prisoners, or how Diplomacy helps the tracking, and it can help the challenge. Stalker's system states that creative use of mental and intellectual skills are broadly useful depending on the situation and players, but physical skills are only good for one thing. It's a baffling blind spot. (not to mention breaking down SCs into three types when there's so much more possibility than that, the best SCs are a mix of a variety of things).
What does or doesn't make sense is highly variable depending on the group. Any system that limits the use of skills across the board short circuits the potential flexibility of the skill challenge system.