Hello Dms.
Seeing the lore discussion and about a dozen others on mechanics, races, rules, etc, I can't help but wonder one simple question - why?
More specifically, what makes you, as a DM, sit down and change a rule?
I feel like each of your questions is extremely deep and the answers required complicated, so I'm going to stop here for now and just answer this first one.
I change rules for a lot of reasons. I think it would be overly simplistic to say that it makes the game more fun, but that's the underlying reason behind all the more specific reasons.
The most common reason to change a rule is probably balance. For example, the rule as written either makes the option too attractive so that everyone is going to be steered to making the same choices or not attractive enough discouraging anyone from making the choice. Balancing rules effectively gives players more freedom and agency while at the same time allowing for more skillful play because there isn't necessarily one hammer you can pound to solve every problem you come up against. Balance also encourages spotlight sharing and team play since no one character has the tool for everything. Ultimately, balancing rules is required of Celebrim's 1st law of RPGs - Thou Shalt Not Be Good at Everything. In D&D specifically, this most often is making spells a little less effective so that they aren't win buttons that make the rest of the party superfluous.
Another very common reason to change a rule is silence of the rules on something that I feel will come up almost immediately in play or which in fact has come up in play. A lot of the time my rules are created directly as a result of finding I don't have a rule for that during play. Rulings are rules. Players all the time make reasonable propositions where the rules don't cover the request or where the rules forbid the request explicitly or by implication even though it is a natural request or where the rules don't deal well with the specific circumstances the proposition is being made in. So a lot of times I'm extending rules to make them more flexible and make clear what the ruling is in circumstances the RAW is silent on.
But this category goes well beyond just having rulings for every situation that comes up. D&D in particular has traditionally lacked whole subsystems for things like evasion and chases, crafting, dominions, etc. What the rules are silent on very often neither the players nor the GM can do, and indeed what the rules are silent on very often the players and the GMs can't do because they never even think about the possibility of doing them. And this category often covers CharGen options which are added in theory by the game makers for the same reason I add homebrewed CharGen options, the silence of the rules on how to create characters of fiction which a player might want but for which the rules provide no options.
Further, this category can include things that I want to be important and specific to my setting, but which the game is silent on because traditionally those things aren't important to the setting. For example, I have explicit rules for divine intervention because unlike most games, in my game divine intervention isn't something that occurs to cover a rare event but something that likely happens a dozen times in a campaign. The gods in my campaign world are very active, and curses and blessings and miracles on behalf of both PC's and NPCs happen frequently enough that they need rules. This category can include monsters that I want in my world but which don't exist in the official rules, or monsters that I conceive differently in my world than in the official rules. And it includes things like naturally sacred ground and holy sites, or the purpose and value of having a roadside shrine.
Yet another common reason to change the rules is verisimilitude. I consider it really important for the success of the proposition-fortune-resolution cycle, that a player that is forming a potential proposition in their head can without any knowledge of the rules develop a sense for the likely outcomes of a proposition. That is to say players need to be able to interact directly with the game fiction without knowing what the rules are and have some confidence that when the fortune test is made the outcome of the proposition will fit within common sense and will fit within common sense at about the probabilities you'd expect. What you imagine as the fail and success cases for the action ought to be what the fail and success cases actually are in the rules, and the rules ought to produce reasonable chances for success or failure depending on what you are proposing to do. As an extreme example, leaping a small puddle ought to be easier than leaping the Atlantic Ocean. I would argue that this verisimilitude test is probably the real reason we bother to have rules at all, because a game system that didn't care about verisimilitude wouldn't actually need many rules.
One of the most complex reasons for changing a rule is addressing rules bloat. Rules bloat come about when makers of a game system recognize the rules limitations of their own rules and begin patching the rules in a peice meal way, often with no more than the intention to sell more rules.
An example of rules bloat common to D&D would be the proliferation of classes and subclasses to cover increasingly small niches. Probably one of the worst rules bloat situations in all of D&D history is late 3.5 edition where CharGen options were sold in every book for economic reasons rather than game reasons resulting in huge numbers of unnecessary and often overlapping classes, which in turn did all sorts of awful things to the game balance. However, a parallel situation existed in 2e D&D with the proliferation of spells to the point of silliness. One of the things I try to do when revising rules for D&D is offer the same number of options while using less words to do it. Sometimes writing rules decreases the total number of rules at your table by reducing the number of rules supplements actually being used at your table. Yes, I do have a 585 pages house rule document for 3e D&D, but this statement becomes less ridiculous when you realize it is a full rules system that replaces all the chargen options in 3.5 D&D while in my opinion allowing even greater flexibility and better balance (eliminating the most broken aspects of the "lonely fun" CharOp minigame all the 100's of CharGen options in 3.5 created). I actually had less rules than most 3.X tables; I just didn't have them spread over 15+ 300 page rules supplements. All the things that were allowed were in one handy place. No need for players to buy $1000 worth of books and combing through them for optimizations, much as I no that in fact some players enjoy that - arguably more than they enjoy playing D&D.
