I do this on a pretty regular basis. Then again I almost never max out my primary stat with point buy, and I play more generalist type characters.
Only if you allow rerolls or other adjustments. With a standard roll 4d6 drop lowest the numbers in most cases will be within a point or two on average. They can always be significantly better than average of course. They can also be significantly worse.
One of the things I have a problem with is that if 3-18 is the normal range of stats for a person (barring curse/disease) then a 4 intelligence for example is not just "slow". It's mentally handicapped and probably not able to function in society without assistance. At a certain point a low stat is not "fluff" as far as I'm concerned.
It just sets up weird things where the guy with the low intelligence and wisdom is played by a smart person and their character is still making brilliant suggestions. I want to allow them to have fun, but at the same hand someone with the intelligence of a baboon should not be a
criminal adventuring mastermind.
But that's just my pet peeve. To me, ability scores are more than just "fluff" and if you don't play that way it's fine.
So you effectively guarantee above-average characters. I just see that as leading to the "5E isn't hard enough" complaints.
But it is better than the alternative. The last game I was in that was "roll for stats" we were all forced to roll (we asked if we could use the Living Campaign point buy instead) and were told no. Then my wife rolled really poorly and was forced to use that character. We didn't last long in that game, which was too bad.
BTW: None of this means you're playing wrong of course. I'm just cranky because I woke up early and haven't had any caffeine yet.

So sorry for the ranting.