I'd be fine with a long rest restoring up to, say, 1d4+CON levels of exhaustion, if folks wanted to keep them as having a possible long-term issue. That in particular would reward Barbarians, since they get benefits from having high Constitution--so a weedy Wizard (-1 CON) taking too much...