Alternatively, "Ok, I was wrong; it can be used out of combat," would also work just fine. And, yes, it is also true that its value out of combat is minimal, but that was not your initial claim.
I know this thread is moving fast, but no, I did not make the simple, absolute claim of universal thruth that Imaro chose to respond to.
Strictly speaking, though, Action Surge is an ability you use on your turn, to take an extra action. You do not have turns out of combat, and can take all the actions you want in most non-combat situations, so, by an utterly-inappropriate-to-5e-philosophy close reading of RAW, you both can't use Action Surge out of combat, and it has no effect out of combat.
That said, a DM ruling that you can expend Action Surge to get some benefit out of combat is perfectly reasonable - as is the DM ruling the same about expending any other resource - and virtually all sub-classes have more, more versatile, and more out-of-combat-significant resources than the Champion/BM fighter's action surge.
I don't much care for RAW arguments, myself, afterall, so the RAW observation was rhetorical - and, yeah, it could've used another 'generally' weasel in there to avoid being trivially 'disproven' by a single counter-example (or the vague allusion to the possibility of one that Imaro uncovered). You could get into a RAW argument about whether AS could even be 'legally' used out of combat, but it's moot in the context of 5e (at least, 5e 'run right,' IMHO - that is, in full-on DM-Empowerment mode), so the real issue is how Action Surge stacks up to other non-combat-applicable features (and, as Hussar illustrated, that's not very well).
So the Ranger and Rogue were cited as being better swimmers than the Fighter....why? Because they likely have higher movement rates? Seems to only be looking at a single aspect of the situation. The Fighter will most likely have a higher STR and is more likely to be Proficient in Athletics...
Higher STR? maybe ... unless it's a DEX-based fighter so it can hold a candle to the Ranger & Rogue when sneaking about...
Same for climbing and jumping. The Fighter will often be better at all of these checks....especially a Champion once Remarkable Athlete kicks in.
They're checks to which proficiency applies, so if Remarkable Athlete does kick in, the Champion will be inferior to anyone with comparable STR who actually is proficient - let alone anyone with Expertise or magic, as well. (But there's an obvious/easy 'fix' for that...)
This is an area that is ignored by those saying Fighters cannot contribute to the other pillars. They can be among the best at Exploration. Climbing, jumping, swimming...Fighters will be better than most classes at these essential exploration based skills.
Those were skills in 3e, they all fall under Athletics, now, AFAIK, so one hit of Expertise, and the fighter's in the dust. And, really, are those the essential exploration skills? What about perception, investigation, stealth, survival? What about picking locks and disarming traps? What about knowledge of nature, dungeons, other planes, and the various dangers you might encounter?
Besides, is getting by without a rope or a raft really all that big a deal?
They also have the option for Proficiency in Perception and Survival, both incredibly useful exploration based skills at which the Fighter can excel.
Which is a nice improvement over prior editions, BTW. Then again, proficiency is readily available to anyone via Background, and proficiency doesn't count for as much as Trained (+5) or 'in-class' (double!) did in the most recent past editions, not without Expertise (double!) or some other feature layered atop it... (BTW, that's one reason making RA stack would be a simple/easy, minor, fix to the Champion's out-of-combat performance, it'd go from half-as-good as proficiency, to proficiency-and-a-half, closer to the Expertise standard of excellence)