The Silmarillion is terrible.
I'm with
@Aldarc and
@niklinna in having a different opinion.
But
@Aldarc's point about authorship is correct - JRRT started writing these stories around the time of the Great War, and then kept rewriting and redrafting, and integrating new elements. The Silmarillion is not a fixed backstory from which the events of the LotR are extrapolated via a process of rational deduction!
I take what I learned about a medieval village and ask what would happen if it existed in a world where monsters roam the wilderness and magic is real and then go from there
I tend to see settings as though experiments.
I've written campaign notes, thousands-of-word treatises on theological commitments of various churches I've invented, etc. I am familiar with the techniques you are describing. I just find your descriptions misleading - because these "thought experiments" are really "inventions", so why do we not use that more accurate label? (In many cases they are also "projections", but given the connotations of that word and the difficulty any given person will face identifying their own projections, "invention" seems more neutral while still capturing the gist.)
For instance, I've read this:
With my own setting, I can take various details from different regions and cultures and combine them in something original that fits the setting and medieval time period. It helps to understand why details developed in the first place in our history as some existed because of specific circumstances.
Elder Anselm is the "parish priest" of Kensla and a major community leader. If things were normal the community leadership would consist of him, the Reeve, and the Baron's bailiff. With the bailiff being the final authority except on matters of religion.
But things are not normal and the bailiff is dead. Depending on how events go, Reeve and Elder Anselm will disagree on what actions to take and this will results in complications for the party.
I attach a summary of the different events that may result from the players undertaking the adventure.
This was a reply to my remark that the setting of the adventure (Scourge of the Demon Wolf) seemed essentially atheistic. And this post reinforces that impression quite strongly. First, there's the notion that the bailiff is "the final authority except on matters of religion" - in mediaeval life there is no such distinct category as "matters of religion". One can try and contrast temporal and church government, but that is also contested to say the least (see eg the investiture controversy). Second, the summary includes this: "Meeting the Elder First: One of two likely encounters that could occur when the characters enter the village, involves going to the church first and meeting the Elder (Priest) of Mitra, Goddess of Honor and Justice." In a context in which there are murders taking place, and dead bodies of tinkers lying around, and bandits in their camp, where are the people doing penance, parading around their village bare foot repenting their sins and praying for relief, etc. More generally, where is the sense of a non-material outlook on things?
This is not to criticise the adventure, which sees a completely normal D&D-esque scenario. Just to point out that describing it as a "thought experiment" or as "based on how mediaeval villages worked" seems quite misleading.
none of that means sandboxes as we've been describing are not driven by a desire for realism
Perhaps they are driven by such a desire. But they seem to be described as if that desire has been fulfilled. They also seem to be driven by a desire for a certain sort of GM authority, which displaces the importance of realism from time-to-time, insofar as players are not permitted to add elements to, or change elements of, the setting in order to increase its realism.
The goal in this approach is a believable naturalistic world. There are lots of methods for achieving that but there isn't a set 'way to do it'. Most often yes the GM will rely on a combination random tables, extrapolation from things that have happened or been established, of notes about the setting (I think importantly though in this style you will rarely see notes about events happening or how the adventure is supposed to pan out as the idea is you are simply providing a setting for the players to explore and interact with), etc.
Though I did notice the following in the summary of Scourge of the Demon Wolf, which does suggest some ideas about events happening and how the adventure might pan out:
Sitting Down at the Kensla’s Tavern: After the characters arrival there will be an important meeting with the villagers at the tavern. This introduces the two main factions (Reeve and Elder) of the village and their issues. Introduces information about the Beggar Clan that is located in the region along with the fact that the Elder of Mitra believes they are responsible for the attacks.
Meeting Yoluf: Yoluf is a local trapper/tracker that the Reeve will introduce to the characters.
Meeting the Beggars: At some point the characters will meet the beggars and learn what they know about the attacks. This encounter could also lead to the discovery of the bandit camp if this hasn’t been found yet.
The Standoff at the Beggar Camp: It is likely that the Elder of Mitra will take matters into his own hands and rouse the village to lynch the Beggars and drive them away.
The Final Confrontation: The characters return to the Wolf Den to fight Arbela and Demon Wolf
The fact that a crucial leadership meeting happens at a
tavern also, in my view, puts further pressure on the idea that this is all based on how a mediaeval village works. I mean, to me it seems that we could replace Reeve and Elder with Sheriff and Parson, replace Beggar Camp with Itinerant Workers' Camp, replace Tavern with Saloon, not even need to change the profession of the trapper/tracker, and reset the adventure in the Wild West. I don't see anything distinctively mediaeval about this adventure at all except for the genre trappings.
Again, this is not a criticism of the adventure. But it is a claim that some of the ways that settings, adventure design etc are being described by you and
@robertsconley are misleading.