That is a great idea! I like it! I think they should probably have a BIG disadvantage on insight checks as well.
Confusingly enough, it can be the other way around. Sometimes. I'm often going to miss "easy" social cues, but I will pick up rediculously subtle things just because I'm more attentive to them.
Analogy time: Completely colorblind person with a spectrometer who's studied color theory. I can't tell that two colors clash by looking at them, because I don't see it, but if you give me a bit of time I can probably make better color judgments than someone doing it by eye.