One of the things I have a problem with is that if 3-18 is the normal range of stats for a person (barring curse/disease) then a 4 intelligence for example is not just "slow". It's mentally handicapped and probably not able to function in society without assistance. At a certain point a low stat is not "fluff" as far as I'm concerned.
It just sets up weird things where the guy with the low intelligence and wisdom is played by a smart person and their character is still making brilliant suggestions. I want to allow them to have fun, but at the same hand someone with the intelligence of a baboon should not be a
But that's just my pet peeve. To me, ability scores are more than just "fluff" and if you don't play that way it's fine.
The veterans at our table often make role playing choices based of what makes up there characters as a whole, including really low stats. A player with a dumb or unwise character doing or saying dumb or unwise things is part of the fun! My dark elf fighter often makes poor choices during combat/role play, even to his own (and the parties) detriment.