It's a tough guy thing in that the barbarian can do it a few levels earlier, exactly as you described. I don't think that's controversial.In what way is this a tough guy thing? A high level wizard or rogue with a decent Constitution could walk off a cliff and survive. You don't have to be a barbarian to pull it off. The barbarian can simply do it a few levels earlier.
So, he's leapt, and he's been thrown from great heights, but you'd assume the one time Conan wanted to cheat on his cliff climbing workout, he'd die?It's not something I've ever seen Conan, an archetypal barbarian, do. Conan might leap off a tall cliff because of reasons, or he might fall, but never because he's simply too lazy to try climbing. That would be absurd and would totally ruin the reader's suspension of disbelief. It's the sort of thing I might expect to read in an exceptionally bad piece of Conan fan-fic. It's certainly not the kind of campaign I want to run though.
This is not a martial/caster thing. This is about not blatantly meta gaming. It's about respecting the verisimilitude of the game.
I don't think calling it a nerf is reasonable. It's not intended as a nerf. It will never even come up for a player who takes a campaign like mine seriously, and plays their character like a living person who isn't indestructible
Whether it is intended to be a nerf doesn't matter. It is a nerf. They are limitations layered onto the core rules and specifically they are limitations that have a greater impact on characters with more hp (martials).
I don't begrudge you your campaign where your players have agreed to these nerfs. Have fun. But ask yourself next time the party faces a giant or dragon or army or archdevil, purple worm, lich or whatever, "Should these living non-indestructible people really have more chance of surviving this than if they jumped off a cliff?"