For me it was the fact that 5E's hard cap on ability scores meant that characters who rolled 16 or better at level 1 didn't have much room left to grow.
Agreed. This is my largest concern with the default 4d6 method. I'd rather have the PCs work up to the pinnacle of their primary ability rather than starting there at first level.