I would have a lessened connection to the world and other characters.
The same as I would if we did the same thing with racial abilities or got rid of classes.
Ah! Now we're getting somewhere.
So, I can see that if we got rid of, say, classes, the game would be VERY different. In fact, so different that I'm not quite sure how to analyze that one.
So let's look at racial abilities: obviously if we didn't have racial abilities, then all of the sudden elves wouldn't be immune to sleep, and halflings wouldn't be lucky, and half-orcs wouldn't have Relentless Endurance (which, let's face it, is awesome) and the human wouldn't be forced to stumble around in the dark because everybody would want torches. So I can see how the game would actually change, in the sense that things would happen differently at the table, without racial abilities.
In fact, let me define it this way: if there were two tables side-by-side, one playing with racial abilities, and one without, an observer would be able to figure out which one was which. It might take a while, or not, depending on circumstances, but eventually you'd be able to see the difference.
(And that's even more true if there were no classes.)
But I still don't understand (yet!) what you are afraid would change with floating ASIs. In other words, how would the observer know which table had floating ASIs and which had fixed ASIs? (Ok, there are all KINDS of cheeky answers to that question, but you know what I mean. I hope.)