The basic point is very simple - if you don't talk like Us, you are one of Them. Human tribalism runs pretty deep. And it isn't just a matter of native English speakers treating non-native speakers poorly about language use, specifically.
Back in high school, I took four years of French. I could carry on reasonable conversation. But trying to communicate with people in Quebec was like pulling teeth. I had better luck using English than I did my non-native-accented French.
You can even see the dynamic between different accents of English, where some accents and colloquial speech patterns are correlated with socio-economic status and political leanings, and you can see strong cases where someone who speaks with the wrong non-local accent can get treated like dirt.
Humans, are, unfortunately, pretty darned disappointing at times.