The d20 roll is random chance, skill, etc. It's not the stat itself. The 6 int person is stupid, but could get lucky, have learned a skill really well, and so on. His actual intelligence is a detriment to any luck or skill he has, because he's dumb.
The idea that the d20 is the stat itself...