I am not a Tolkien expert at all, but actually this feeling I'm talking about in this thread came back to me once more after watching the latest Hobbit movie, which gave me a very different feel about demihuman races.
"The Battle of the Five Armies" -> Elves, Dwarves, Hobbits and Orcs are not as rare as provoking awe: everybody seem to take their existence pretty much the norm. But all of them live secluded somewhere else, and stick to their kind most of the time, except in times of war against common enemies. Furthermore, looking at the size of the "armies", it seems possible there is not that many of them in the world.
I think you're right about this. Tolkien goes to great lengths to commission a world where the
differences between the sentient races have been a cause for enmity and fear (rather than, for example, the Star Trek idea of IDIC). That's not to say that Tolkien's races aren't designed to work in harmony--they are--but the Melkor/Morgoth influence on the world means that even
within a race there are splinters, rather than combines. Hence, different nations of men, different nations of elves, and different clans of dwarves.
"Dungeons and Dragons" -> everywhere is full of Elves, Dwarves and Halflings (but also Gnomes, Half-orcs, Dragonborn, Tieflings... for fear than any player's PC is treated unfairly?). Even an apparently small percentage as 5% of race X in a town is actually a lot more than in Tolkien. It seems to me there just isn't anybody living within other races territories, just very temporary visitors at most.
There are also strong tendencies in D&D to "humanize" non-human races as much as possible, for instance by assuming all of them have authorities, economy, laws, technology, professions etc. similar to humans. I am not sure, but I didn't get the feeling that Tolkien reveals much about those aspects, and if you don't reveal/explain too much then you can actually imagine they are completely different.
Tolkien does go into some detail on this, but as you have intuited, I think he goes to some lengths to alienate the races from one another; he describes them
both in terms of their similarities to humans and in terms of their differences. I think he hinges a lot of it on the lifespan of the various races. By granting elves literal immortality (i.e. no natural/old-age death) he sets them wildly apart from short-lived humans. Dwarves are an interesting case in that they straddle the line: long-lived by human standards, but still incomparable to immortals. Hobbits...well, hobbits seem to be a less developed idea (ironically). They do live longer than humans, but not by much?
At any rate, the different life-spans create different lifestyles. Some off the cuff thoughts on this:
Elves have forever to do things, so they tend to be meticulous, serene, and aloof. An elf's life is worth a hundred of any "lesser" race's lives because a single elf can outlive all one hundred of that lesser being's lifespans stacked end to end. Elves have the capacity to look at other race's lifespans the way a human looks at a fruit fly: they live too short a time to
really develop into "personhood". (Not that this is the default elven view, but that they have the
capacity to see it this way.)
Dwarves have centuries to build, perfect, and train. They're not a "chosen" race and have none of the hopeful qualities associated with humans or elves. Their values are ingrained in them: a love of beauty (that can turn to greed), a love of strength (that can turn to power-lust), and a love of kin (that can turn to xenophobia). Add a few centuries of life to each of those values and you get traditions that cover every aspect of dwarven life.
Humans are the odd ducks in that they (a) die and (b) that death is considered a "gift" because they'll have some strange role in the world-to-come (whereas elves and dwarves will not). Their short lifespans make them jealous/admiring of the grandeur and depth of culture in other races. Human nations that survive tend to emulate the traditions of the elder races but with distinctly human twists.