For the demi-humans I believe it is because they started off as nabbed from Tolkien where dwarves are Jewish Vikings, Halflings are little English farmers and gentlemen, and elves are Cate Blanchett templated. So not a lot of color in the base stock there. This gets compounded by the fact that Tolkien/Monster Manual demihumans already have fairly defined white ethnic subsets broken out for them (high/wood/mountain/wild elves, hill/mountain dwarves, hairfoot/stout/tallfellow haflings). Throw them into Greyhawk and you get an innovation and subversion, the partially norse Dokkalfar inspired dark elves being literally black skinned to get the ball rolling in the 70s with the GD series.
Oriental Adventures gets hairy Asian dwarven Korobokuru in the 80s and Forgotten Realms then has a hinted at southern dwarf population in its
1e campaign setting, who later get revealed as the dark skinned gold dwarves.
For the most part the settings start out as mostly fantasy Europe standard with demihumans thrown in, sometimes with a little renaming for the setting. Greyhawk innovates drow and duergar, FR starts with the Monster Manual/PH/Tolkien base and then starts to expand with more everything. Even in
Dragonlance, where they put Black Humans explicitly into the setting, demihumans were still just mostly renaming the base White D&D demihumans (Qualinesti/Silvanesti/Kagonesti Elves, Kender Haflings) without messing around with ethnicity beyond blue aquatic elves and skinny hobbits.
As others have cited before, Harfoots like Sam are said to be darker of skin. I love Sean Astin's portrayal, but the character was miscast.
There are many FREE peoples in Middle-earth who arguably could and should have been depicted as darker of skin, including much of the population of Gondor (The subjects of the Númenórean Realms-in-Exile were quite mixed descent, and there's an argument to be made that Númenóreans themselves looked altogether more like Egyptians or at least olive-toned like much of the Mediterreanean coastal peoples than the white northern Europeans they've been shown as often). Don't get me wrong - the 2nd House of the Edáin (House of Hador) SHOULD be blond and fair of skin, and this carries over to SOME Númenóreans and definitely to the Men of Rhovannion and their descendants (the Lake-Men, the Wood-Men, the Éothéod of the Upper Anduin, and thereby the Rohirrim). But the House of Béor (1st House) was darker of hair and appearance, yet we treat them as if they're also super pale, when that's just not in the text. In fact, Beren Erchamion, the most famousest of Line of Béor, is contrasted against the silvery arms and shadowy hair of Lúthien Tinúviel, suggesting that his dark hair was not as contrasting against his skin as Lúthien's was against her.
Tolkien also did not have Cate Blanchett to work with when he wrote the book. Galadriel was based on Mary, mother of Jesus (Tolkien's words) - though there are also elements of the more complicated and perilous Lady of the Lake from Arthurian legend in her; see the throughline from Tolkien's Breton/Kernow Arthurian
Lay of Aotrou and Itroun to the discussion of Galadriel in the
Unfinished Tales.
In any case, Tolkien's elves were largely rooted in Germanic and Celtic faerie tales - there's a direct throughline from the Peter Pan-esque Tingfang Warbler and the Victorian-era pixies of
The Cottage of Lost Play in the
Book of Lost Tales and the taller Álfar/Vanir/Túatha Dé/Tylwyth Teg-like residents and the Elves of the final books. This is why Elves are mainly white in Tolkien, because he saw them as the Pre-Germanic peoples of the British Isles (Insular Celts and their forebears) from the perspective of putting together an Anglo-Saxon-Jutish mythology for Great Britain. He was never attempting to tell a fantasy story for the whole world, just trying to right the single injustice of the loss of the English mythological tradition. He certainly wouldn't approve of D&D lifting his free peoples of Middle-earth straight from the pages for repurposing, just as he didn't understand the appeal of his books in America audiences (though he happily took the money, because he needed it and was willing to compromise these principles for the sake of putting his children through school).
Finally, while Dwarves are very clearly inspired by the Israelites and European Jews in
The Hobbit in the sense that they're a people with a promised kingdom that they're in exile from, and speak a Hebrew-inspired language, and have many of the visual stereotypes of the racist caricatures made of Jewish people in Europe, these belie the development of the Dwarven people in Tolkien's mythology. In the beginning, they were scarcely more than a inserted Svartálfar/Dvrgr from the Norse Sagas - down to their villainous role in the earliest stories he wrote of them. The Hebrew-based grammatical structure Tolkien used for Khudzul is a late-Lord of the Rings creation; even in
The Hobbit, the Dwarf-runes were just Anglo-Saxon runes transliterating modern English for ease of cypher.
The Hobbit began Tolkien's shift on the dwarves as a narrative tool away from antagonistic to interesting and complex culture, one that Tolkien literally went on a journey with Bilbo from discomfort and rancor in the beginning (where their instrument summoning and musical choices is almost more reflecting of Corriag from
The Mabinogion than anything from Snórri!) to saying goodbye to dear friends, welcoming the survivors always for tea should they be passing through the Shire. There's an argument to be made that
The Hobbit is in many ways about building up those stereotypes to begin with and then dismantling them over the course of a journey on the road, much as any good war story might (you latch onto bad character traits of your fellows-in-arms at the beginning of the march, but by the end you're the best of friends and realise those were YOUR projections, not their inherent bad character). We see that the Dwarves are no less greedy in their disposition than the Elves of Mirkwood or the Men of Lake-town, and the three peoples nearly come to blows over it, but all put aside those differences because of a common threat in the Goblins and Wolves.
Tolkien certainly was drawing upon those visual characteristics - WHICH SHARE RACIST TONES - because he was drawing on the visual and literary history of Nordic Dvrgr and was using them as shorthand. I would argue that he was drawing on depictions in the history that had long applied dwarven iconography in a racist manner to Jewish members of the community. But by the time Tolkien was linking Hebrew with Khudzul, he had abandoned the idea of a villainous, greedy Dwarven people. In his letters he always spoke fondly and in defense of the the Jewish people. It IS true that by his
Later Silmarillion, he had long since linked the peoples in his mind, and the creation story of the Dwarves in the published mythology is not too far removed from the
Akeidat Yitzhak (the Binding of Isaac). These are present perhaps more in the final
Lord of the Rings Appendices, Unfinished Tales, and
Silmarillion texts, but are less present directly in the more dwarf-relevant
Hobbit. We're just intimately familiar with the comparison due to adaptations which have drawn on the scope of the people from the peripheral texts.
Dwarves certainly weren't Viking-like - that's more of a Peter Jacksonism, especially once John Rhys Davies decided to give Gímli a Scotsman accent (remember that the history of Scotland is a history of conquest and integration - the Albidosi (Prethin/Picts/Northern Brythonic peoples), the Old North Cumbrians, the Roman garrison men between the Walls, the Dal Riáta Ulstermen, Viking petty kingdoms, and finally Anglo-Saxon and Anglo-Norman encroachment and integration into the United Kingdom).
I'm not saying Tolkien wasn't racist - he had a lot of unconscious biases that bear out throughout the texts, and these should be addressed. But we shouldn't lay the problems of D&D So White at Tolkien's feet necessarily. Rankin/Bass, Bakshi, and Jackson all have depicted in their films a Middle-earth FAR more white than Tolkien did with his prose.
