Don't get what bonus, the stat bonus? So effectively, in easy mode, you are only proficient in your classes skills, can try anything else at +0. In standard mode, obviously ability to specialize in a skill and lose proficiency in one you had (or skill points or something). Again, you can always try something, but a +0.
Effectively, a dexterous cleric +3 is as useless at "Dex" checks as a clumsy one -3 ... hmmm, not liking. Could have major pain points in multi-classing.
But, just therory-craft; will wait and see