Not every martial character takes Mage Slayer, even though they could (assuming Feats are available in their games). This implies that they don't want the ability that badly- other things take priority, even for Fighters and Rogues who get extra Feats to work with.
If Mage Slayer was given to every martial, every caster would likely want the Feat to counter it (I can't imagine why they wouldn't, at least). This would make it a Feat tax. Feat taxes are bad for the system (especially so long as Feats are considered optional content).
From this, it may be possible to conclude that Mage Slayer is far less valuable to martials than "anti-Mage Slayer" would be to casters. So you'd be giving martials a small buff in exchange for giving casters a larger nerf.
I don't think this would bring the classes into balance, since the whole point of this house rule is to achieve that. Put another way, how often do you encounter foes that Mage Slayer would be useful against?
It's campaign-dependent, but there are far more monsters and foes who can't cast spells than those that can. What this house rule is looking at is buffing all monsters and players by giving them Mage Slayer.
It's a pretty big paradigm shift against the PC's, in exchange for a small buff. Because D&D is a team game, and making it harder for a party's casters to use their magic to aid in battle makes the game that much harder for martials- don't be fooled into thinking this only affects some players!
Also, that buff will become even less impactful if 2024 monster design follows the trends seen in Monsters of the Multiverse, where "spell attacks" and not actual spells become more prominent (this will affect Counterspell as well).
I know the prevailing thought is that 5e is too easy, so I'm sure a lot of people won't think that making the game harder for players will be a problem. But nothing in the game is built with the idea that spellcasters can't cast spells the vast majority of the time.
In fact, if anything, you want casters to cast spells more often, not less, so you can drain them of their resources. It's been my experience that players aren't going to cast spells if they think they'll lose them- I saw this in AD&D where even a small amount of Magic Resistance made magic-users loath to use their spell slots.
Further, very few players open themselves up to opportunity attacks if they can avoid it. Especially arcane casters, who have less hit points than other characters. So what happens when having a single enemy next to a caster effectively turns them off until someone has to come kill the monster for them, which might open them up to opportunity attacks in kind?
The way I see it, either your groups already use tactics that prevent enemies from getting close to their pointy hats and very little changes, or the game grinds to a frustrating halt as a play pattern 5e wasn't really built around is enforced.
There's a lot of nuance that comes along with this houserule that has to be accounted for.
Alright, said my peace, you can commence with disagreeing with and dissecting my point of view now.