Your Absolute Worst Down Under...

[Lancelot]

can the races intermingle with one another and have offspring using real gene tables? with dominant and recessive aspects of life? Or is it a sort of caste system similar to the covenant from Halo where they're born into life?

just thinking outloud here, something I might add in my next campaign (With beholders a warrior caste, like military tactician types)

It's a caste thing, although there is an implication that some of the koroashi races might have resulted from genetic modification by the culture-as-a-whole in the past. For example, the kopru may have been (willingly) mutated from land-bound mind flayers through genetic modification in the far distant past. However, even the koroashi themselves might not know the answer to this any more.

In any case, the castes aren't hierarchical. Each of the "tentacled" races has their own accepted role in the culture. The kopru are not subordinate to either the aboleths or the mind flayers, for example. The leadership of the koroashi is a great unknown. Elder Brains, perhaps? Or maybe it's a colossal mind-linked democracy. Or maybe they are all servants of some unknowable Great One that dwells outside of mortal comprehension. None of my players know, and none of them really want to know. Aboleths and mind flayers scare the living @#$% out of them.

Note also that the caste method avoids the mental image of aboleths and mind flayers mating or using implanted tadpoles to produce some of the other races. I briefly considered something like that, and lost several nights sleep. The number of times I woke up screaming: "Oh god, the horror! The madness! The tentacles! Ia Ia Illithid Fhtagn..."