Continuing to use the term Gypsy for themselves is for them to decide, not for those of us who aren't. We should be respectful of the people who say it's a racist term and not use it except in the very specific cases of people who individually use it.
It's like terms for older organizations like NAACP and UNCF. They continue to use older terms that are considered offensive now because they worked at the time they were created and named themselves. They are not for polite conversation now.
So I'd suggest you stop trying to argue that the term Gypsy isn't broadly considered racist and inappropriate. It is.
I'd say "gypsy" is less like the NAACP and UNCF, and more like the term "queer". Yes, it was a pejorative at a certain point in time. But it's also commonly used today as a neutral identifier and even a term of pride. In the USA, at least.
If someone asks you to not call them "gypsy" or "queer", by all means you should stop. But many people wouldn't consider it inherently racist. Of course, context matters.