That's a good point, but it raises an important question. Specifically, that question is why do these discussions pretend that the same does not hold equally true for social & exploration challenges,An often overlooked issue with Martials vs Casters is that the game expects you to win the majority of combats. Martial Characters help you "Combat Better", but do you really need to Combat Better when Combat Fairly Average is usually enough to succeed at the combat pillar? Certainly you lose sometimes, characters can die, and campaign goals can be failed because you have to back off and rest too often. But the default is winning. Over and over. And Martial characters primarily help you win more at the thing you are already winning.
This is also true for Exploration to a lesser extent, but there are far fewer Exploration specialist classes.
If challenge A could be trivialized by spell A if that spell is an option and that spell is prepared or someone else could just spend a bit longer physically solving it without expending any resources. An example of that alternative method might be using something like the 50foot rope in burglar dungeoneer & explorer packs alongside athletics or acrobatics what gain does doing it a couple minutes faster when there was little if any time constraint to begin with. What is gained by expending a spellslot to save a couple minutes that weren't being counted?