Apologies in advance for the wall o' text.
But, it's not "every time"! It's arbitrary! It's the DM telling a story.
This, I think, is where you and I differ. When I DM, I'm not "telling a story," which implies I have more of a right to dictate the narrative than the players do. I'm providing a setting for my players, and they're free to do what they want with that in terms of narrative. In the former approach, you're deciding what would be "cool" or "flashy" or whatever and improvising ad-hoc mechanical rulings to make the mechanics fit the narrative; in the latter approach, I let the rules dictate the outcome of player actions and narrate what the outcome is.
Making arbitrary rulings of this sort (A) takes some of the narrative control out of the hands of the players and (B) doesn't always (or even most of the time) achieve what you want to achieve. For instance, to continue with the same example, let's say you rule that a righty attacking a lefty or vice versa should take a penalty on attack rolls due to the difficulty of parrying left to right. On the one hand (no pun intended), this takes away from the character to some extent--what if the character trained with a left-handed master, and so has no problem parrying lefties? What if the character, in fact, has an easier time parrying lefties because he trained exclusively with left-handed masters? Sure, in general PCs won't have a background specifically mentioning this, but it can make a difference (what if they run into their master later in game and the DM or player decides on the spot that he's a lefty to pull an Inigo Montoya "I am not right handed!" trick?), and if they do mention it and make a point of this, you have to make an ad-hoc rules change to your ad-hoc rules change. This particular example is fairly trivial, but the fact remains that ad-hoc rulings can have wide-ranging, often unforeseen, consequences.
On the other hand, the ruling doesn't actually work from a verisimilitude perspective. If anything, lefties should take penalties against other lefties, because they'd have been trained to fight right-handers and wouldn't know what to do against another lefty. Thus, a ruling intended to make the fight more "realistic" has in fact done the opposite. You don't always have the time or ability to reality-check these kinds of things mid-combat, so while they may seem to work at the time to you, (A) the players may think "Wait a second..." after the fact and possibly be upset if the ruling had a large impact on the battle and (B) people who are knowledgeable in the area your ruling covers (such as a fencer for the lefty/righty example) would derive less satisfaction from what they perceive to be a nonsensical ruling.
It's akin to the DM saying, "You notice a strange design on the goblin's shield. You saw it before on the mage's robe." That's the DM arbitrarily deciding that the character noticed and recognized the design, and put two-and-two together rather than making the character roll some sort of INT/WIS/Perception check to see if he noticed the design.
It's very close to the same thing, except we're talking about combat rather than exploration.
And why is that an example of a good ruling? Serious question.
There are two schools of thought to D&D, generally speaking: challenge the player or challenge the character. In the AD&D days of yore, the structure of the game challenged the player. The player handled RPing, so a low-Cha orc fighter played by a thespian was more persuasive than a high-Cha thief played by a recluse. The player handled trapfinding and puzzle solving, so if the player didn't think to check the lock for traps, oh well, bye bye PC. In this sort of circumstance, it should really be up to the
players to remember this sort of detail; if they don't realize that the goblins are connected to the mage's guild, that could make later battles harder as the goblins are reinforced by the mages and the PCs don't know to take the mages out.
Nowadays, the philosophy is to challenge the character, with 3e providing skills for social situations so an 18-Cha bard is a smooth-talking scoundrel whether played by an actor or an introvert, skills for trapfinding and -disabling so that the player doesn't have to read the DM's mind to have even a chance of survival, and so forth. In this sort of circumstance, it should really be up to the
character to remember this sort of detail; that's what Int and Wis checks are for, to see if the savvy character who lives in this world and is concerned about mage/goblin relations remembers the connection even if the player forgot, or conversely to see if the slightly-dim character forgets the connection even if the DM would have otherwise reminded the player. You can't completely remove metagaming from the equation, obviously, but the rules for Knowledge/social skills and Int checks and such are meant to help with that.
Again, if the DM just decides that the characters remember or know something, it (A) is just as arbitrary and fair as deciding that they
don't remember or know something, which is detrimental to the player and (B) takes some of the responsibility of the players/characters away (and privileges, as if the DM says "Sorry, I'm not going to remind you of X" a player is perfectly justified in saying "My character is a 20-Int rogue,
he'd remember that!").
This makes combat exciting because of the arbitrariness. In a regular game, without this aspect I'm speaking of, the character doesn't care as much that his toon was reduced to 2 HP. My arbitrary call makes the combat so much more memorable. "Hey, remember when Thrallan blocked a guy and got his fingers crushed, then swung his leg out and kicked the guy in the balls! Man, what an encounter!"
Regular RAW dicing back and forth does not allow this element of the unexpected--the mystery--the risk of combat.
If a player can embrace this type of thing, he just might find the game so much more fun.
First, and I mean no offense by this, I'd like to point out the irony of someone who advocates realism and immersion in their games and then calls a PC a "toon."
Second, in my opinion, combats shouldn't be memorable due to DM calls, they should be memorable due to player actions--again, the players should have investment in the narrative as well. "That battle where I hit the guy and he dropped his sword and I killed him when he went to pick it up" is not nearly as satisfying for the player as "That battle where I disarmed the guy and I killed him while he was defenseless." One involves the DM deciding that the enemy loses his weapon and the PC getting an advantage with no particular input on his part, the other involves a PC deliberating making the choice to deprive his opponent of his weapon and capitalizing on the success of his own tactical decision.
The "regular RAW dicing back and forth," as you call it, is not merely soulless calling out of damage numbers and hit point totals. Positioning is important, creative use of resources is important, negation of enemy advantages is important; there are many other things in 3e combat besides attack rolls and AC. In fact, when playing by the rules, a player can be confident in his tactical decisions. "I charge down the hill at him and attack!" he says, knowing that he'll get +1 attack for high ground and +2 attack for the charge and thus that this intuitively-advantageous action is in fact mechanically advantageous in the game and worth his time to do. If the DM can at any time say, "You know what, the hill is kinda slippery, make a DC 9 Balance check or you fall" or "You're kinda out of control running down a hill, take -2 attack," then players can't make decisions that are as well-informed as they could if they could rely on the rules, and would in general be disincentivized from making creative attacks (unless you ad-hoc most attacks anyway, in which case combat is a crapshoot).
Embracing and trusting the DM is the hard part for many players, I think.
I'm a DM, not a player--I DM the vast majority of the time, and haven't played a full campaign in a year at least--yet I disagree that this is about trust; I've been playing with one of my groups for over a decade and the other for three years, and my players in both groups trust me implicitly as a DM. However, they trust me for several reasons: They know that I abide by the rules for the most part, and can thus be trusted to be impartial and not "out to get them." They know that if I'm going to make houserules I declare them before the game starts and any on-the-fly rulings I have to make I'll get their approval on, and thus I can be trusted to be fair and not pull rules changes mid-session. If I were to one day declare "Okay guys, I'm going to start trying something out to make combat interesting" or just do it without telling them, they'd probably go along with it, trusting me to do a good job and make things more fun...but that doesn't mean that it's okay to do this if your players trust you, nor does it ensure that they will continue to trust you if you do this.
If your players enjoy arbitrary combat tweaks, that's great, more power to them and I congratulate you on finding a group that meshes with your playstyle, and I'm not at all implying that making arbitrary rulings will suddenly make your group hate you or anything like that. However, the fact that this works for your group does not imply either that it would work for others or that it is in fact a better system than the default rules.
It does have to do with trust. I once ran a game where the players wrote down their stats, class, level, and equipment. The bard asked what he should write for abilities, and I told him not to write anything down. When asked what he could do, I told him "bard-like stuff." We played the entire game with only a pool of d6s. Players would say "I want to swing at the undead wyvern's throat." I'd say "roll 1d6" or "roll 3d6" or whatever felt right. They'd roll, I'd call the events. The players felt extremely immersed in the game. They didn't even know the rules. It was all arbitration, and I made all the "mechanical" decisions. If I felt something was within their power, I let them have it. If it wasn't, then I didn't.
If you don't mind me asking...if you're going to make things up as you go along, why play a rules-heavy system like D&D in the first place? And if you weren't attempting to simulate D&D with that game, why do you think the same system would work for rules-heavy D&D as did for a rules-light freeform game? Again, the D&D rules are fairly comprehensive (some would say too comprehensive) and consistent; people generally play rules-heavy games because of this fact. Were I to run a lighter system like FATE or Fudge, or one of the many quicker and more abstract indie games, or something freeform like Amber Diceless, you can bet I'd have all sorts of on-the-fly rulings based on player descriptions, and the game would be better for it. That sort of play works for those sorts of games, and in fact is practically the whole point. However, in those games the DM is given many more blank spots in the rules he's expected to fill in himself, and the combat systems are basic enough that such ad-hoc rulings are easy and beneficial.
As far as a shared storytelling experience... well, I'm very against the narrative style of play. That's not to say I'm against plot or setting or political intricacies or anything else. I'm very into a story being discovered by player and GM alike. However, the GM, in my opinion, deserves much more control over the story than the players. If nothing else, a huge perk of being the GM is letting your creative juices fly, and that includes weaving an interesting and enjoyable setting for the character.
As I mentioned above, as a DM I see my job as world-builder, not storyteller. The DM has created an entire world, has countless NPCs at his metaphorical fingertips, and has the control of the general plot. The DM's job stops there. the PCs are the only characters the players control; they should have as few narrative constraints as possible. The exact events of the plot are the only impact the players can have on the world; the DM should interfere with the nitty-gritty details as little as possible. So while I completely agree that a DM should be able to create an engaging setting, memorable NPCs, and a compelling plot, when combat or a social encounter or a bit of exploration comes up he should sit back and let the PCs direct the course of the encounter without trying to tweak things to make things turn out how he thinks they "should."
Nope, I wouldn't allow my players to name their bonus. But again, I'm pretty against a narrative style of play. However, your "if you let players decide, you may as well not have rules!" is an extremely weak argument. Do you think rules should matter? Then you better not houserule anything ever, and you better not ever let someone do something not described in a book, or you may as well get rid of all the rules!
However, as I do consider myself in charge of the game (it's up to me to keep the pacing reasonable, the story and plot and setting interesting, and it's up to me to tell the players how they fair when they try to kick some butt), it's really up to me how something happens. I try very hard not to make arbitrary decisions, but it's still me making the decision, not the players. If I decide that attacking from a 1.5-ft. high platform doesn't count as high ground, then it's my call. If I said it did, it'd be my call. The difference between a circumstance bonus for high ground or for parrying is conceptually nil.
I try to follow the RAW as closely as I can, but I definitely break them to follow the RAII (rules as I intended... you see, I wrote the book we play from). To that end, if I say there's a certain circumstance bonus or penalty in play, it's not a "right" or "wrong" call objectively. Subjectively, it definitely is. Don't play it if you don't like it, for sure. I totally agree that it sucks if you play with it, if you dislike it. And again, it's amazing we can have totally different play styles, and each get our way. The hobby truly is amazing in that regard
There is a middle ground between pure RAW and arbitrary rulings. That middle ground is the DM and players working together. When you make houserules, lay them out upfront, before the game begins, and make sure the players agree with them or tweak them if they don't. When you make rulings during play, make sure it seems fair to the players. D&D isn't exactly a democracy, since the DM's word goes, so when exercising that power a DM should make sure it results in fun for everyone involved. Sometimes you have to make a call the players disagree with because of hidden factors or other considerations (e.g. the "No, that doesn't count as high ground" ruling above) but your rulings should be consistent and fair so that it's easier to trust you when you do that. And that's perfectly okay, the DM has veto and ruling powers for a reason, but making a habit of arbitrary rulings is, I think, a mistake.
And, for the record, I agree completely with your philosophy that the rules take a backseat to fun; I simply believe different levels of DM intervention in the rules lead to that increased fun than you do.