Woah. My initial thought is I wouldn't take a 3, ever, for any reason.
Yep. No way.
Mechanically, a -4 modifier on any stat (even the least-relevant stat for your chosen class) is way too punitive to justify the extra +1 or so modifier on your primary stat over something like point buy, the standard array, or an average rolled stat array.
There's really no such thing as a true "dump stat" in 5E that you can afford to have a 3 in. Every character needs every stat at some point, not only because of all the various skill checks that could come into play, but especially because of the various ability saving throws for spells and effects. There are certainly stats that are less of a priority for each class, but I wouldn't want to have less than a -1 or -2 modifier to any stat on any character.
Besides, roleplaying-wise a 3 stat means that personal is basically profoundly handicapped in that aspect. This is well beyond just "noticeably bad at it".
D&D Stats in Simple Language | DMing With Charisma has a nice breakdown of the correlation between stat numbers and actual manifestation. The descriptions for such a handicap of a 3 in a stat are as follows:
STR: Needs help to stand, can be knocked over by strong breezes; Knocked off balance by swinging something dense.
DEX: Incapable of moving without noticeable effort or pain; Visible paralysis or physical difficulty.
CON: Frail, suffers frequent broken bones; Bruises very easily, knocked out by a light punch.
INT: Barely able to function, very limited speech and knowledge; Often resorts to charades to express thoughts.
WIS: Rarely notices items, people, or occurrences; Incapable of forethought.
CHA: Minimal independent thought, relies heavily on others to think instead; Has trouble thinking of others as people.
So for example, a 7 or 8 INT means they're stupid or uneducated, whereas a 3 INT means they are practically non-verbal and barely capable of performing basic personal caretaking functions without assistance.
A 7 or 8 DEX or STR means they're clumsy or weak. A 3 DEX or STR means they are nearly bedridden.
So I can't foresee anyone with such a serious hindrance in one aspect of their life becoming an adventurer.