For me, it's a bit of a balancing act. Obviously, it'd be unreasonable to expect, and I know I don't expect absolutely everything you do to be straight out of a book somewhere. Fantastic eldritch machines that power floating cities are cool, and we probably don't really need to know the exact area or amount of weight they can support. On the other hand, something like the "spell you've never heard of or seen with no roll made that's astoundingly effective" that I've run into a good few times - both in personal experience, and in some printed sources like the Avatar Trilogy modules that bridged 1e and 2e FR - instantly disengages me from the game and annoys me with the story. I personally lean much more towards wanting rules and stats for everything, because I'm a worldbuilder and a sandbox GM much more than I am a storyteller, so I address things from "does this result in a fun world structure that isn't laughable from a 5000' view, or maybe even a 2000' view" perspective.
For example, the Nine Hells in 3e were pretty laughable from a 5000' view. Asmodeus managed to rule them unquestionably. However, a lot of deities made their homes in the Nine Hells. By their written statblocks, even the god of kobolds could appear directly before Asmodeus at any time and nearly unfailingly slay him in a single blow. As such, the world structure was difficult to take seriously, because it failed to work within the rules presented.
A town of regular humans without any food supplies or trade would be pretty laughable from a 2000' view, as would the existence of tribes of kobolds whose weakest members were CR 19 who did not gravely impact regional politics, high-level (12+) brigands fully decked out in magic items and preying on 1st-level travelling commoners, et cetera.
I don't care quite so much about the sea-level view. I don't study how landmasses are formed, or study medieval economics, or whatnot, so I don't worry too hard about the economic model of the game working for commoners or whatnot. I just care when the error is so blatant that I can see it from 2000 feet up.
While the kobolds and the Nine Hells structure both fall into these sorts of errors, the former being bad worldbuilding and the latter being bad for worldbuilding because of its place in the assumed cosmology of pre-4e D&D, the main objection is to things like the latter where, without extensive writing, a being cannot actually do the things it has been described as doing or is described as doing, because it is a combat statblock with a description stapled on.
