Because removing buffs isn't a harmful act. It's aggressive for sure. But harmful? Pshh it couldn't hurt a fly.
Harmful don't need be necessarily agressive. Removing buffs targeting a creature (It isn't willing creature) and requires ability check is obviously a harmful spell.
But, anyway, the DM will give the final word of what he considers Harmful or not.
Whatever, It isn't a threat.