Let's look at the biggest influence on modern fantasy: JRR Tolkien. The goblins there didn't resemble Jewish stereotypes. In fact, he modeled the dwarves after those stereotypes so we know without doubt that he did not model the orcs/goblins after them.
I wasn't talking about Tolkien specifically, and his work has its own problems.
Firstly, depicting the dwarves as semitic received its own backlash.
Secondly, Tolkien described orcs as "squat, broad, flat-nosed, sallow-skinned, with wide mouths and slant eyes: in fact degraded and repulsive versions of the (to Europeans) least lovely Mongol-types." So now they're anti-Asian stereotypes.
And in those books, the goblins did pretty much everything the nazis did (limited to technology) and more, because they also kept their captives to eat. They invaded lands, murdered everyone they could, captured and tortured prisoners (same basic thing as as a death camp), etc.
That describes
most warfare throughout history,
including cannibalism. You are trivializing the atrocities committed by the Nazis.
The goblins aren't any worse than the Visigoths, the Mongols, the Aztecs, the Spanish, the French, or the British. They are hardly in the same league as the Nazis, unless you want to argue that all of those nations were as bad and/or worse than the Nazis... which both trivializes and normalizes the Nazi death camps.
More people are more upset about fictional monsters being treated as wholly evil and worthy of destruction, but we don't see hardly any of the same outcry when it's barbarian tribes in a historical wargame being destroyed.
Firstly, yes we do get upset. People were livid about
Scramble for Africa because it trivialized the colonization of Africa, resulting in it being cancelled.
Secondly, wargames don't depict barbarians as "wholly evil and worthy of destruction." The wargames don't endorse genocide.
You might feel differently if you were a third-generation expat living in Japan and Japanese people treated you like you were a character from their pseudo-European fantasies.
Firstly, that's a problem with society being racist. To that I respond with this video:
Secondly, I don't think that's a good reason to discourage everybody from writing fiction about cultures other than their own. That would logically accuse the overwhelming majority of fantasy fiction as cultural appropriation, because most fantasy authors write about cultures other than their own.
George R.R. Martin, a white American, has appropriated numerous Native American, European, Asian, and African cultures while creating
A Song of Ice and Fire. Should he be discouraged from writing any fantasy other than that which concerns white American culture like baseball and apple pie?
Mizuho Kusanagi, a Japanese woman, appropriated ancient Korean culture when she wrote
Yona of the Dawn. Should her books be banned because she isn't Korean?
I strongly disagree with discouraging authors from writing outside their culture because that is both censorship and racist. Amélie Wen Zhao, a Chinese-American, had her reputation smeared during the publication of her debut novel
Blood Heir because she tried to tackle human trafficking (based on her experiences as a Chinese national) without being an African-American woman.
Is it even possible to create a fictional group in a setting that someone can't correlate to a real life group?
I don't think the problem is that there are often parallels between fictional groups and real groups. I think that's going to be inevitable when you're depicting fictional groups as essentially human with qualities added or subtracted.
I think we should strive to avoid depicting fictional groups as caricatures, especially if they have any similarity to real world stereotypes.