Why not!1st part of Mage slayer needs to be general rule and with advantage.
rest can be paired with +1 ASI for half-feat.
also add, when you critically hit a caster with AoO while caster is casting a spell, caster loses a number of spell slot levels equal to your proficiency bonus. Casters choice.
I would not play a caster with your house rule, but you are free to try it with your players.