This is a good idea on the surface (and shows your DM can spot problem areas) but it doesn't interact well with rules for hunger, thirst, environmental effects, etc. You shouldn't be able to starve yourself to death, have a meal every week or two, and be in ship shape.
Now, you can make special rules for how Exhaustion works with those conditions...but at that point it's turning into a lot of new rules and becoming its own mess.
Yes, they cancel them out and make you gain little benefit from your class features, because now you are functioning in those areas like someone who isn't a raging barbarian.