It makes us unknowable to each other.
At the heart of the constructed view of the world that generates the "cultural appropriation" meme, runs the highly racist idea that each person is completely unknowable to each other. That idea hides behind words like "diversity" and "authentic". It shows up in ideas like: "You just can't understand what it is like to be a woman." or "You can't understand what it is like to be black.". It seeks to exaggerate and highlight our differences rather than what we have in common. It emphasizes not that we are more alike than different, but that we are more different than alike.
Nineteenth and twentieth century racialists (not all of whom were necessarily promoting racial hatred, even if hatred was the inevitable and most visible consequence) used to argue that there was an essential national character to each race of mankind which was inescapable. You could take a person out of his environment, but his inherited national traits and tendencies would still be inescapable.
Now the inheritors of that same logic no longer speak of inescapable national traits, but will speak of an inescapable and unique racial experience which everyone on the basis of their race will still have. It amounts to the same thing in new language. There is this idea that a person with dark skin, raised in suburban New Hampshire to a well to do family inherently has by virtue of their skin color an 'African' experience which is distinct from and unknowable by anyone who hasn't the proper melanin and that on that ground alone has more in common with other dark skinned people than he does with his fairer skinned peers that he ate lunch with in high school. Conversely, a person of fair skin, who grows up in Africa, speaks African languages, and who wrecked his car by hitting a Kudu, because of his fair skin still has a distinctly 'European' experience which makes him unable to understand or share in darker skinned experience.
The idea ultimately is that we are so different because of our race or what ever group we are said to belong to, that it is impossible to walk a mile in each others shoes. Whites inherently have privilege and cannot truly understand oppression or speak of oppression authentically. Blacks inherently are victims and cannot truly speak from a place of privilege. No person can truly understand anyone else, because there is no common human experience. No one's experiences crosses the lines. No one is able to imagine on the basis of their own suffering the sufferings or joys of others. Even the claim to empathize with and share a common experience with someone outside or your group is called hate speech.
This theory has it that an inner city white kid inherently has more in common with a rural white kid or a suburban white kid, than he has with inner city peers of a different race. Suburban black kids have more in common with inner city black kids than they do with their suburban peers. In short, your race remains your destiny under this theory, and diversity is counted not according to life experiences, viewpoints, philosophies or anything else, but can be tagged by race alone.
Ultimately, this new racism is the same as the old racism. The same stereotypes are in play. Black suit wearing businessmen aren't authentic voices; real authentic black voices are criminals. "White men can't jump". Mexicans like spicy food. Asians are goofy glasses wearing dorks. And so on and so forth. Only this actually demeaning stereotyping isn't counted as damnable "cultural appropriation" as long as "authentic" "well-intentioned" people do it, where as non-hateful exploration of other cultures is - regardless of your intention - accounted hate speech.
Lying behind all this lie is a bit of Newspeak twisting of the word identity so that it means both itself and its opposite. We now have people using the word "identity" both for the distinctive thing that makes you unique, and for the non-distinctive thing which makes you a member of a class or group. When we identity people now, we know longer point to what makes them an individual, but to a long list of group memberships as if merely knowing something about ones group membership told us more about the individual than knowing the individual did.
Why should being black be like anything, and really is it like anything? Just as intelligent black man could be dismissed by 19th century racial movements not as disproof of the theory that blacks could not be intelligent, but simply as an outlier that didn't represent the overall pattern of his race, so modern racists dismiss men of the same character - or really anyone of any group that doesn't fit their stereotype - as being inauthentic and non-representative. Today we've reached the point we no longer are pointing out the outlier to dismiss the theory, because the theory has become undismissable and unquestionable. Today the outlier is celebrated precisely to mark them as an outlier, and when that person says anything remotely to question the theory, this is proven by the fact that they are easily dismissed as inauthentic.
And they'll do this even if the person unquestionably shares the life experiences that supposedly make the group unknowable and distinctive. If you lack the stereotype, if you don't mouth the expected rote opinions, you aren't an "authentic voice".
But it's noteworthy who the "authentic voices" are said to be. If you were cynical you'd say that the "authentic voices" are being carefully chosen by privileged groups to further their own privilege and influence.
My brother was closely acquainted with Comanche medicine man. Full blood. Not this 1/16th crap where 15 of the persons ancestors were white and yet he thinks he's got exclusive title to be the spokesperson for native culture. You'd think this would be an unquestionably authentic voice of his people. Regarding sports teams adopting names like Braves and Indians and the like, his opinion was that the White people were counting coup. That they defeated the red men in battle, and so it was to be expected. If they had won, they would have done the same thing out of very much the same impulse. Indeed, you can see the same impulse when native Americans put on the jackets of the US Cavalry soldiers. His idea was that the white people didn't count coup on us because they despised Indians. There is no honor in counting coup on something pathetic. They counted coup because they wanted to share in the bravery and strength of their enemy. That isn't to say that he thought there weren't real injustices and real problems out there, but he was convinced this whole thing regarding sports teams was a controversy imposed on the Native Americans by people who wanted to make themselves feel good and not something most of them really cared about. And if you go looking, you'll find other voices in the Native American communities saying much the same thing. Why don't we engage those voices? Are only angry voices worth engaging, and only if they are angry about the things we want them to be angry about?
At best, "cultural appropriation" is about trivializing justice down to holding a few easily held opinions about things that don't matter much, and then feeling good about yourself - like some Ave Maria repeated endlessly to absolve yourself of not actually changing anything. At worst, it's the stealth recreation of every important point of 19th century racism either by the cultural inheritors of those views or perversely by the inheritors of the people who had those views imposed on them, clutching the very demeaning views to their chests as if they were literal truths. Because it's there 'identity' or something.
I'm inclined to think it's a lot of both.
You note later: "I merely made the point that white nationalists have easily appropriated parts of this argument and I think there is a good reason why: it helps create more barriers between cultures and people."
You are mostly right except for one point. The white supremacists didn't appropriate parts of this argument. They invented it. The relationship between the ideas is the other way around. The SJW's appropriated this argument from the white supremacists. It's fundamentally accepting the world view of the KKK or the Neo-Nazi's as correct, and then in the midst of that hateful world view trying to construct some sort of moral framework. But it fundamentally can't work, because the framework itself is hateful and twisted. The ultimate end of pushing this crap on people will be to increase hatred and feelings of alienation and to try to push people into tribes. Because once you deny that people have a basis for understanding one another, and deny that they belong to a common community, then you've basically denied that they have any reason to care for one another. That's ultimately the reason why if you look not even very hard at the SJW community, if you even only look as mainstream as say The Guardian, you'll find that preaching hatred of the other is conventional within the community.