Honestly... I kinda love it. Though I think, maybe, the best way to do it is to make it ability specific. You're using your highest attribute to overcome the effect.
Physical attributes, Str, Dex, and Con, gain Fatigue. Mental attributes, Int, Wis, and Cha, gain Strife.
That way the penalty is...