Lanefan
Victoria Rules
If you're playing/running Isle of Dread as a one-off adventure then sure, what you say is absolutely true.But starting Isle of Dread with "You're sailing from X to Y and then a storm blows up, and beaches you on this lonely island . . ." isn't worldbuilding. (Hussar's post indicates that this isn't the canonical way of starting X1. But it is a possible way, which is enough for my point.)
Which is [MENTION=22779]Hussar[/MENTION]'s point. The Phantom of the Opera happens in Paris, but we don't actually need to build Paris; we just allude to it. The GM can narrate that the PCs are sailing from X to Y without anyone actually having to build X or Y. They are just names in a bit of introductory framing.
But I'm assuming in all cases here that these adventures are embedded in an ongoing campaign...which in this case means the PCs were in X for a reason (even if only to get on the ship to sail to Y) and are...were...intent on going to Y for a reason. That they were in X means I've had to narrate at least a bit of description about the place; and whatever Y is would have to have had some sort of narration or description or clues because otherwise why would they be going there?
We're also told that it's far from the nearest settlement and away from travelled routes, strongly implying that the PCs will not be easily able to go back to town and resupply and-or recruit new characters to replace their dead. Because of this, and because low-level D&D play tends toward resource management, a DM is going to want to know how many days it takes to travel from town to the adventure site so as to monitor the PCs' food supply...which makes positioning both the dungeon and the "nearest settlement" on a map a rather useful thing to do. Noting any intervening terrain features that may help or hinder travel would also be useful. It's also trivially easy to do these things; then later as the campaign goes on they can be expanded upon until soon enough you've the genesis of a game world.Here's the "game world at large" for B1 (it's p 6 of my PDF version):
area, Rogahn the Fearless (a fighter of renown) and Zelligar the Unknown (a magic-user of mystery and power) pooled their resources and expertise to construct a home and stronghold for the two of them to use as a base of operations. The location of this hidden complex was chosen with care, since both men disliked visitors and intruders. Far from the nearest settlement, away from traveled routes, and high upon a craggy hill, the new construction took shape. Carved out of the rock protrusion which crested the heavily forested hill, this mystical hideaway was well hidden, and its rumored existence was never common knowledge. Even less well known was its name, the Caverns of Quasqueton.
The fact that there is a largely unknown (in terms of existence and name) stronghold constructed in an isolated, craggy, heavily-forested hill, is not worldbuilding. That's the barest of flavour. The only actual bit of the world that's been "built" for this adventure is the caverns.
Me, I just prefer to move that work from within the campaign to before it starts. That way I can think through the in-play implications of what I've designed* and tweak it to suit.
* - a process I manage to mess up at every opportunity, but hey - live and learn...

Lanefan