It wouldn't solve the main issue I have with it, which is lack of realism. People can't select the stats they have. Instead, they're born with whatever stats they have and make do. Plus, it isn't all about high stats for a lot of people who roll.
I've always assumed that the people with low or substandard stats stay at home on the farm. After all, adventurers are the equivalent of a team of professional athletes, not the couch potatoes who watch them.