I mean at 3rd level a Barbarian can rage 3 times, meaning you could Frenzy 3 times. Each time you use Frenzy, not only do you make yourself weaker, but you impact the whole party. How frustrated would the party be when you are consistently moving at half speed, suck at Athletics checks you're supposed to have advantage on, and have to attack recklessly just to negate your exhaustion? It's one thing if the penalty only affected the barbarian. But by making the barbarian incrementally weaker in a way that can last multiple adventuring days, you are also making everything harder on your group. D&D is a team game. So those choices to accept exhaustion don't just impact you.
I don't exactly have fun when I'm playing with someone who needs to be carried (literally at times) and can't do the job they are supposed to. If you're a tank, and you can't tank without negating your tankiness, you suddenly have the power to unilaterally decide for the group if everyone has to play on hard mode.