Okay, but what about them being able to outright cast Beast Sense, Speak with Animals, and Commune with Nature? Albeit as rituals only, but the Magewright NPC from ERftLW also can only cast leveled spells as rituals, yet nobody denied that they are magic.
You and
@Hussar are right about them being able to perform a ritual version of
Commune With Nature to speak with an animal spirit. But you could also write that off in-character as a Drug Trip on some Ayahuasca or similar.
As far as
Speak with Animals, there's always Mick Dundee, who no one thinks is actually magic, just VERY Australian.
Beast Sense, though, is 100% Beast Masterish.
Though he could also speak with animals, it feels way more "Disney Princess" than actual magic.
Point is, based on how you play it out you can be pretty much a Martial character with Totem Barbarian as long as you play it that way. And there's no material requirement (or ability) for Totem Barbarians to use magic to fight. Which is what a Martial Barbarian cannot do to maintain the "I'm just a big strongboi who bashes things" that was outlined in the OP.
You -can- play up the Totem Barb as mystical AF in combat, going full Vixen from DC Comics where you tap a magic necklace and the Bear Spirit appears behind you, jumps in your body, and makes you super durable. But you don't -have- to.
You can just be a big strong character who has a particular style of fighting based on animals that makes them super duper cool beans awesome without any magic whatsoever. Though they have the option of doing some shamanistic magic rituals outside of combat if they -want-.