And this is why fiction is loaded. You are loading assumptions into the word and it has everything to do with style. I would argue, the stuff that happens at the table matters but so does the stuff the GM prepares. If the GM decides "This castle is going to exist in this spot, no matter what" it exists in the setting, whether the players find it or not (and that is important because it should exist in the setting in a sandbox whether they find it, they don't, they find it in session one, or they find it in session 10, and treating as existing matters because even if they don't directly encounter it, they may encounter signs of its existence-----if there are encounters in the area around the castle, very possible those encounters are inhabitance of said castle for example---even if the players don't realize that until ten or twenty sessions later)