Or in other words 'do their jobs'.
It's not that it 'doesnt matter' for casters. It' that martials get EVERYTHING from attack rolls and skill checks and movement. That's what they do. Every single round, they're making multiple attack rolls, being targeted by multiple attack rolls, and when not in combat they rely on skills.
A caster gets his potency from casting spells that force other creatures to make saving throws. Movement penalties matter less when you can teleport, or fly. Skill checks to do things matter less when you can cast a spell and avoid that skill check (or simply get your familiar to do it for you).
You could literally play a Wizard to 20th level, never make a single attack roll, and kill more things than your party Martial.
So you're going to force the PCs to push on with an adventure, with the martials at disadvantage to attack rolls (denying Rogues sneak attack) and disadvantage to all skills, and at half HP and movement?
Meanwhile the casters can Fireball, Hypnotic pattern, Magic missile, Heat Metal, Animate Objects, Wall of Force, Dominate person, Hold Person, Sleep, non attack roll damaging cantrips etc etc etc and able to Teleport via Misty Step etc with impunity?
Who would you rather be playing in the above; the Wizard (disadvantage to attack rolls, half HP, disadvantage to Skills) or the Rogue and Fighter?
Who does the rule hurt the most?