Ok, from here on out, I am defining world building as what is contained in Chapter 6 of the Third Edition Dungeon Master's Guide. If it is contained within those specified pages, then I consider it to be world building within the context of this discussion.
I suppose you'll respond with some further semantic nonsense about the LIKELIHOOD of something appearing in a gaming session. This will raise questions like, "Which gaming session, exactly? Does it need to be the very next session? Or is it okay for me to prep a few sessions in advance?" And, "So it's a matter of the probability of something appearing? How probable does it need to be, exactly, before it becomes setting instead of world-building?"
Please, look up the definition of a spectrum. If you do so, it will answer all of your questions quite nicely. I have stated, time and time again, that there is NO CUT OFF POINT! There is no point where red becomes orange. If you know where that is, you better be applying for a Nobel Prize, because no one else in the world can. At one end of the spectrum, you have elements that will always feature in an adventure and at the other end, you have elements that will not feature in the adventure.
The Five Shires are detailed in X1 The Isle of Dread. However, they are not once referred to in the actual adventure. No NPC comes from there and there is pretty much no way to get to the Five Shires from the Isle (without a whole lot of travel time at least). The details of the Five Shires is completely and utterly superflous to the adventure. That's world building.
The central plateau where the Black Pearl rests is key to the Isle of Dread. It's the setting for the climax to the module. It's referenced multiple times throughout the module. That's pretty much as close to setting as you can get because you cannot complete the module without going there.
However, and I've admitted this multiple times, there are lots of things in the middle that may or may not be needed. I tend to err on the lighter side of work because I'm lazy. Some may go the other way. However, there becomes a point where the details you create are not going to factor into the module without some major arm wrestling on the part of the DM or rewriting the adventure itself.
I hope this becomes clear because I don't know how else to say it. There is a spectrum, a continuum, a range with Keep on the Borderlands on the one side and Realmslore articles on the other.
These questions will stymie you. So you'll either (a) ignore them; (b) claim that these questions constitute ad hominem; (c) attempt some unpredictable semantic nonsense to avoid the issue; or (d) pout for a bit and then repeat your initial assertions as if they hadn't been thoroughly discredited a dozen times over.
Please, before you try again, open up your DMG. Take a look at
what the creators of 3rd edition considered to be world building. Think about how that relates to the idea of world building vs setting. Take a deep breath or three. Come on back.