Really?!
I think that's an absolutely convincing argument, because the degree to which it's "even more" unrealistic is pathetically tiny. So I honestly think you might want to muse on that one a bit longer. If the degree which much, much larger, you could make a logical argument. But going from -2 to 0 in D&D rules? No. Being concerned about that is pure aesthetics.
Sure it's "movement in the wrong direction", but it's extremely small movement, and to make halflings "realistic", you'd need fairly complex rules with special cases for different sizes, but if we're going for that kind of realism, we also need to make it so humans, no matter how heroic, basically automatically lose any kind of melee vs Ogres and larger creatures - the only remotely viable way to bring them down being ranged weapons (spears, bows, etc.) or specialized weapons (extremely long pole-arms etc.). If you insisted on sticking to existing rules, you'd probably need to give halflings like -6 STR and cap them at 4 STR or something. Would you actually propose doing that though?
Also you'd probably want to cap elves at like 14 STR, and so on. And uh-oh, here comes Realism's creepy uncle, "Sex-Based Stat Modifiers", who has been living by himself in a basement since 1979. Nobody likes him, but when you get obsessed with "realism" in species stats in a fantasy game, he hears a clarion and call and bursts out of his basement at a run!