There is a lot going on here. I don't know that I'll respond sensibly to all of it, but will try to convey some of the thoughts I had.
You often respond more sensibly to what I write than I do! :>)
(2) Anyone could have been Obi-Wan's apprentice: Luke's name is Skywalker; his father was a great pilot and a great Jedi who left him a lightsabre as an heirloom; his father was killed by Darth Vader. These are the things that make Luke the appropriate heir to the Jedi tradition, and cast him in dramatic opposition to Vader and the Empire. (In a very different way from Leia, whose opposition is political, not personal/spiritual in the same way, at least before the Death Star is used to blow up Alderaan.)
Yes, but as we now know, Vader killed a lot of Jedis. It didn't have to be Vader's son that became the hero (although that's the direction the story later took). In fact, one could argue that the entire Star Wars story shifted from a pulp/action story to something with more depth with that one line at the end of
Empire. The series shifted from rebellion to redemtion.
There are multiple alternatives to random determination: railroading is one; what I have been referring to as player-driven RPGing is another.
A living sandbox is another. I'd say random determination is between 10-40% of what's going on at any given point. Although many of those randomly determined things are options for a placed encounter.
But consider using (say) the Stormtrooper massacre at the farm as a framing device. I think it would be harsh for a GM to just do that willy-nilly! It makes sense, rather, as the consequence of a failure. (Say, a failed Navigation roll to travel from Obi-Wan's house back to the farm, having realised the threat posed by the Empire's hunt for the droids.) But to know that that would be an appropriate consequence for failure, one needs to understand what is at stake for the player in declaring that action for his/her PC. This is what I'm saying I couldn't judge without knowing what is motivating the action declaration, which was the trigger for this particular discussion.
So part of the problem I'm having, is that when I look at a scene (like this) I read it as I would run/experience it in my game.
The "failure" on Luke's part was to purchase the wrong (right) droids and follow the one that ran away. Those were the actions that led to the massacre at the farm.
If I'm the DM of that game, the stormtroopers are hunting down the droids, and killing all they encounter to avoid witnesses that the Empire was there. (To be honest, other than a plot element to free Luke from his home, I'm not sure why the stormtroopers killed the jawas or his family, as it's clear that the Empire is already on Tatooine once they arrive in Los Eisley). In any event, that's what the stormtroopers are doing, so if Luke's not there (by design, or he'd be killed too), they are.
The motivation of the action declaration? Again, the movie unfolds like a game session:
1) Uncle Owen (NPC) has dragged Luke along to get a droid that can help on the farm. Mission: Find droid.
2) DM describes scene at the jawa crawler. There's a bit of role-playing in selecting a droid. It smokes, the Luke PC picks out another one that the DM (as 3PO) points out as another suitable droid.
3) Return to home/bed/morning where the DM describes (via the NPCs) that R2 has disappeared. Luke's motivation? Find him. Could be others, but that's the only one that matters. Off he goes. (Naturally, because the adventure won't continue if he doesn't).
4) The DM describes the terrain, following the tracks, scenes with Obi-Wan, etc. until we get to Luke realizing that the stormtroopers would be heading to his home when they find the jawa crawler.
So I guess what I really don't understand is how that would play out differently if the players were writing more of what's going on in the world. How would this scene be authored differently at your table? The lines seem to be very obviously drawn at certain points in terms of authorship.
(4) The writer provides the world and the characters react to it: the writer of a fiction is a person in the real world; the characters exist only in imagination. So the two can never interact.
Of course they do. The imaginary characters react to the imaginary world. Whether it's the DM describing the world, or one of the players, it's ultimately the same thing:
One of the participants is describing what's going on, and then the rest of the participants react to that.
Whether it's one participant responsible for all but the actions of the characters (DM), or that's somehow divided amongst the group as a whole is the only difference.
The writing of a fiction includes the writing of the actions of persons who are elements within that fiction. The question about RPGing is who gets to write which bits, according to what sorts of procedures. Eg does the GM frame Luke (as a PC) into a confrontation with the Empire because the GM wants to run an Empire-oriented game, or the table as a whole does, or the player has written something about opposition into the Empire into the PC's backstory, etc.
The same sorts of considerations apply to the farm massacre. Who wrote in these family members of Luke the PC? Is the killing off of the farm providing the player/PC with something that s/he wanted (freedom from farming) though at a high cost (death of family members)? Or is it undesired through-and-through?
These are all questions about authorship, not about in-fiction events.
And as I've said, the players get to write their backstory, with my setting certain boundaries. They'll give me their backstory, and I'll let them know if anything is out-of-bounds, and usually give them some ideas of my own (which are almost always tied into the world). They work those together, and it's up to them if they like what I propose or not and incorporate it if they do. Sometimes we'll go back and forth a bit. But in the end, they have as much freedom as I can give them in the setting, and they have final approval for anything that falls within those boundaries. Obviously, I'm aware of these motivations, goals, etc. at this point.
So the player would have detailed the family and the farm. The killing of the family is an event. In my campaigns it would have been an event dictated by the course of action in the story created by the players - that is, Luke purchased the wrong (right) droids, which then put him, and his family, in the path of the Empire that is looking for the droids and killing all witnesses.
As the DM, that course of action is put into play from the goals of the Empire, the mission of the stormtroopers, and the fact that the trail of the droids leads to Luke. How Luke feels about it is irrelevant in regards to the Empire and the actions they take.
Once the event has occurred, you're asking if there is a silver lining - that's not my decision to make. The player decides how Luke feels about it. He might share how he feels, he might not. Obi-Wan has to react to it, but he reacts to it in character, that is, only with the information at hand, what he can see, and what Luke does and says. If Obi-Wan feels compelled to ask about it, then I'd ask about it, but in character as Obi-Wan. Otherwise, the next decision is still up to Luke - are you with me or not?
Certainly, the more the player reveals about their feelings, the more it gives me to work with. But my knowing the character motivation isn't a requirement. There's no known character motivation in many published adventures. They often give several hooks to help the DM pull the characters in. Sometimes there are specific ones, but then you have to ensure somebody fits that mold. Otherwise the motivations are usually the same - kill the evil monsters, get treasure, save who/whatever.
In many campaigns, they never get past the kill monsters, get treasure (and gain levels and abilities), as the basic motivation. I like to tie them into the world better. They are people, who would prefer to have more than a peasant, probably have some family, probably eventually would like a family, a home, stuff they like, etc. In fact, that's really my primary focus initially - to point out to the players that the characters are people and should act like it. Once again, though, that's instruction and insight provided outside the gaming table, not during the session.
If Luke decides he's free from the farm and his Uncle, might have an inheritance, and go off to university with his friends, then that's where the story leads. Of course, the droids are his too, which might lead the Empire back to him again.
Does the GM frame Luke (the PC) into a confrontation with the Empire? Well, I make it possible. I create the story arc - that is the plans, goals, and actions the NPCs/organizations/monsters/world will be taking unless something changes it. In this case the stormtroopers are after the droids. If Luke doesn't pick these droids, then the stormtroopers find them with the jawas, kills them, and takes the droids. Meanwhile, Luke is still farming. Not as interesting a story perhaps (at least so far), but as I said, it's the players that are writing the story. It's certainly very likely that his path will cross with the Empire's, but it doesn't have to.
In my campaign, the PCs are actively involved in 14 different story arcs right now, if I recall. Some of those are individual personal ones, others are group ones. And some of the group ones are shared by a few of the characters, some by all. They've probably completed as many, and ignored probably 10 times that by now. Because I don't want to be reliant on them picking the "right" one, nor do I want to set up the scenario where they are picking one particular option because they know the DM needs them to.
Luke has to buy those droids. That's really the only option in that story.
In addition, while I think it makes sense and is prudent to have several choices lead in the same general direction, to the same long-term possibility, I don't think all choices should lead there. But I also want them to learn and see, at least sometimes, that their choices have consequences. There is rarely a "right" choice. For example, you know that there is a hit out to assassinate 3 people. You don't have time to alert the authorities, and you can't save them all. There will be consequences related to the one you save, and consequences related to the two you didn't. You might not always see them, but I'll consider them anyway in case they might come into play later.
To what I am seeing more and more as my "simulationist" brain, this is what makes the most sense to me. When you walk out your door in the morning, you could meet somebody you haven't seen in years, you could be in a car accident, you could witness a crime, or there could be an act of nature. You have no control over any of that and, for the most part, none of that gives a crap what your motivations are for the day. The car accident doesn't care if you were on the way to save drowning puppies, or off to rob a bank. All the accident cares about is that you happened to be in that place at that time.
My dramatic fiction brain thrives on seeing how people react to the world around them. Particularly in difficult situations, moral dilemmas, matters of life and death, things like that.
I frame the situation: Stormtroopers killed your family.
The next question is, "what do you do?" and not, "how does that make you feel?" The players handle the emotional and psychological reactions, along with their actions. Usually all three of those are fairly evident at the table).