A setting like Middle Earth only has a half-dozen types of non-humans (elves, dwarves, hobbits, goblins, orcs, trolls) and they are both isolated and exceptionally rare...
Contrast that with Eberron, who has several integrated societies where over two dozen different species...
To me, neither Middle Earth nor Eberron exemplify the issue in the OP, because both settings give guidelines for how different races can interact. In Middle Earth the rules are stricter and in Eberron the rules are more lenient, but there are reasons and guidelines for both that allow you to make a group that has some level of cohesiveness.
D&D 5e, OTOH, just gives you a literal demon race (tiefling) and a demon powered class (warlocks), and throws them right alongside a traditional white knight (paladin) and holy messenger of god (cleric). With no justification, guidelines, rules, or anything. 5e just shrugs its shoulders and expects the DM to accept that as a baseline level cognitive dissonance. The gonzo nature of of the "circus troupe" the OP suggests isn't inherently about diversity. It's about a lack of reason.
That being said, this does start to push into the question of "What should be the base setting for D&D?". Which, IMNSHO, is a question WotC has gone out of their way to pointedly not answer. I think as D&D gets more and more options while refusing to define a baseline setting, increasing the level of gonzo in the baseline is an inevitable side effect.