I usually agree with you, but not this time I'm afraid.
I'll start with two assumptions: 1) that the game system being designed for includes divine powers/beings/etc. that characters can or are expected to interact with (without these, cosmology becomes largely "fluff" for characterization purposes and doesn't need much design behind it), and 2) that the PC-accessible parts of the setting in question are bigger than a single small area (e.g. designing other worlds or planes is irrelevant to something like a Blades in the Dark setting where the PCs don't really travel outside a single city, but knowing what deities are worshipped in that city might become important in play).
Given those, I posit the opposite is true: you have to spend time and energy on the cosmology. More precisely, you have to either start with the cosmology and work down or work the cosmology in as part of every other phase of design, because everything - everything! - in the setting ultimately flows from its cosmology and-or its deities and not working them in at every phase risks an end result where things appear bolted on rather than seamless.
The overarching conflicts in a setting - in D&D these would often be Law-v-Chaos and Good-V-Evil - either stem from its cosmology or are mirrored by it; and these conflicts drive everything from character design to (sometimes) species design to elements in the physical setting. Middle Earth is another example: the overarching conflict in that setting, mostly reflected at a mortal level in the LotR books, stems from a dispute between deities told of in the Silmarillion. If Tolkein doesn't design that cosmology side-along with the rest of the setting, the story wouldn't make any sense.
Also, entire species may - or may not - exist in the setting based on the desires or whims of some deity or other.