I just don’t understand why, if you want to roll ability scores, you would want to reduce that random element. If the goal is to produce more balanced results, why not just use point buy? If the goal is to increase the chances of getting higher stats and producing more heroic characters, why not just use point buy with more points? If you want random stats, what is the value of intentionally limiting the variation?
I doubt if you asked anyone I play with they'd ever say balance was a factor. In fact they'd probably look at you oddly. Some would even say something insulting like "If you want balance, go play 4e.".
Of the umpteen different methods presented in in 1e, 4d6-1 was just the one that seemed to please most of the people at the time. We liked it better than the straight 3d6 we cut our teeth on in Basic/Expert, it was still pretty random, & it took less time and fiddling than a lot of the other methods. And it allowed us enough of a chance to roll up the stats needed to have things like Paladins, Rangers, Caveliers, etc in our games without cheating.
And over the years, as the groups expanded, shrunk, shifted, etc? It just got carried over as pretty much our default year after year. Eventually the game caught up to our default....
As for PB? We don't use it because:
1) Our core group doesn't enjoy fiddling about that much to generate the stats.
We've allowed for PB & Array mostly for new/other players who join us. We can accommodate them. What we won't do is change how we generate our stats.
Thing is? The people who've joined us? They see us not being fussed by high/low #s, weird variances in character stats, etc shrug & roll 4d6.
2) Standard PB does not allow the full 3-18 range. And to make it do so requires more fiddling than we care to invest in.