Sorry, I'm not trying to mock, but that rationale makes more than a few presumptions, such as that the Wizard is practicing sword play at all.
If nothing else, there's a baseline assumption in class-based games that a character will use all of their abilities at some point. That's why your skills improve at all, even if you don't get XP from using them; the assumption is that you're using those skills in proportion to how many monsters you defeat (or how much gold you find), such that we only need to track one variable (monsters, or gold), in order to make generalizations about all of your abilities. Your character shouldn't even have abilities, if they are never going to use them.
Maybe that's an unreasonable assumption, but ignoring that assumption adds a lot of complexity to the mechanics. I know that third edition had a suggestion in the DMG that the DM should only let a PC gain ranks in a skill if they actually used that skill since the previous level, but I also seem to recall that the advice wasn't repeated in Pathfinder (in spite of it being the same exact game with the same exact assumptions of play), and my only explanation is that they figured it wasn't worth the effort to track.
That leads into the issue of role-playing in order to bypass mechanical restrictions, which is more evident with the simplified weapon proficiencies introduced in third edition. In second edition, your fighter could take proficiency with longsword or battle axe, but a third edition fighter is automatically proficient with all martial weapons. And it's not the case that every single fighter ever has actually trained with longswords and rapiers and battle axes and glaives and heavy flails and so on, as much as role-playing your fighter as a weaponsmaster who
had done that would not be mechanically unbalanced, so they just handwave it all rather than requiring every fighter to explicitly have done so.
If your objection is that a wizard shouldn't get better with a sword unless they actually practice, and your objection can be overcome by the player remembering to say that they practice swordfighting every night, then the alternative case (where the wizard doesn't practice swordfighting) is not worth modeling mechanically.
And there stands the difference in our positions, in a nutshell. You see it as a problem that a Wizard isn't also a minor duelist, and I don't.
Is your objection to all class-and-level games, then? Because every such game that I'm aware of - certainly every edition of D&D, at least - has had spellcasters automatically advance with their weapon abilities (combat matrix, THAC0, BAB, level-bonus, or proficiency bonus) regardless of whether they actually used them.
That's a different question than whether advancement rates should be symmetric or not. If you think wizards shouldn't advance at all with their weapon skills, then that's understandable, especially from a verisimilitude standpoint. Especially in 5E, where everyone has cantrips, it's fair to assume that a wizard never actually swings their sword.
If you're willing to assume some amount of advancement with abilities that aren't explicitly used during play, though, then we can move on to the question of how much. (If you're not, then fine, but you should be objecting to the whole class-and-level system in that case.) If we're going to say that the wizard gets better with their sword over time, then how much better should they get? Should they improve much more slowly than the fighter, such that we're technically tracking the math even though it will never come up during play? Or should their relative disadvantage remain constant, such that there's a chance (however minor) that they might actually use that ability, and the math doesn't collapse into the inadequacy of d20 granularity when they attempt to do so? From a perspective of gameplay balance, the answer seems obvious.
