I note also, once I know that a given bit of terminology upsets someone, the practical reality is that if I keep using it, I'm making the decision that I'm willing to upset them, and also to communicate to them that I'm willing to upset them. That doesn't mean it's necessarily wrong; I can conclude that their response is not enough to justify whatever downsides I see to changing my language. But it is a thing I try to be aware of, and conscious of, and not handwave away. I don't necessarily object to choosing to do something that results in people being hurt, but I do object pretty strongly to handwaving it away or ignoring it, because declaring that hurting people doesn't matter is usually not a step towards a better and healthier society.
I don't disagree with you here. One of the things I am objecting to is the inverse; hand waving away the responsibility of the other party.
Also, that there are often other factors beyond just someone's feelings; it's often not just about the cost of changing your language from a personal perspective. Changing your language could also signal other things on a broader level, or imply tacit approval to something you don't agree with.
An easy example is: if someone says they're offended by any mention of or display of homosexuality.
The key issue there isn't that it would be inconvenient for you to censor yourself from such mentions. It's that it would be wrong to do so. Make sense?