That's a really easy position to take when you're not on the receiving end of it for years and years and years.
I obviously disagree. When someone is telling you, directly, that what you are doing is hurting them and they don't like it, that doesn't make them the a-hole. It makes you one for not stopping. There's really no other solution here. Telling someone, "Oh, it doesn't really hurt you, so, I'm not going to stop" makes it all about you. It is telling people that your convenience is more important than considering their feelings.
Well, you'll keep hearing it until you stop hurting people.
I mean, what else is there really to say? You have been told that what you are doing is hurting someone. Instead of stopping, you are insisting that the person telling you that they have been hurt prove that they've been hurt. And then questioning that explanation and refusing to accept it unless it meets your criteria for "being hurt".
At what point do you think you come out of this on the right side of the issue?