Some thoughts...
First, as mentioned before, Detect Magic definitely needs to be on their list of spells. Better yet, allow them to detect magic when they use their Divine Sense.
Secondly, I don't generally see too many NPCs using concentration spells specifically. Make it broader by allowing them to prevent an enemy struck from casting spells at all. Or censure them, making them take some damage every time they cast a spell or maintain concentration on a spell.
The Oath of the Ancients Paladin also gets an anti-magic aura. I'd just re-use the wording.
Finally, I'd put Spell Interruption on a short rest timer and possibly make it a reaction. Being able to shrug off spells every once in a while is one thing. Being able to cast Dispel Magic/Counterspell pretty much at-will seems a bit much. Then again, I doubt the DM will generally want to ever waste even a single NPC action trying to target a character with this feature with a spell, making it kind of moot.
Other features that could be interesting. Taken largely from the anti-magic Templars of Dragon Age and Protectorate of Menoth from iron Kingdoms...
Passive bonus like advantage on attack rolls or bonus damage when the target that just cast a spell or is under the effects of a concentration spell. Could apply to summoned monsters as well.
The ability to treat weapons as magical for the purposes of overcoming damage resistance. Or ignoring all defense bonuses/resistances that come from spells when attacking and dealing damage.
The ability to turn demons (possibly fey), since this trope of the Holy Anti-Arcane Knight usually works under the assumption that arcane magic involves dealings with malevolent spirits. Although again, it'd be very similar to the Oath of the Ancients.