Ability scores. And I don't mean that the specific ability scores used by D&D bug me. (They do, but that isn't the point.) I mean that the whole concept of ability scores bugs me.
First: The numbers you use at the table--skill bonus, attack bonus, save bonus, damage--are composites of other numbers. Any change to those "source numbers" ripples across your entire character sheet. This makes chargen needlessly complex, makes game balance harder, and steepens an already-harsh learning curve.
Second: Along with class, the ability score serves as a broad-strokes description of your character concept. This often forces players to choose between "my ability scores and class describe my concept" and "my character is good at the things my concept says they are good at."
If my concept is "cunning warrior who uses brains over brawn," that suggests a battlemaster fighter with high Int and modest Str. But all of the battlemaster's combat tricks rely on Strength or Dexterity, and care nothing about Intelligence! I am not good at the things my concept says I should be good at.
Get rid of ability scores, and there is no conflict: I just make a battlemaster and it works as advertised.
Third: The distinction between "ability score" and "ability score modifier" is a layer of stupid frosting on a bad design cake. Getting rid of this distinction wouldn't fix the deeper problems with ability scores, but having the distinction--where you almost don't care about the base score, but once in a while you do--makes everything that little bit worse.