Lots of great responses and I appreciate it, but still absolutely confused why so many DMs are afraid of improvising during combat to improve the narrative; and so much so, its considered immoral? Do you really not trust yourselves to make the right decision, ever?
I've been doing this like 40 years now and I can confidently say that reflecting on the times when I fudged things, I made things worse more often than I made them better at least 2/3rds of the times.
As a counterpoint, I have been improvising during combat regularly for several decades and cannot think of a time where it hasn't worked out for the better or at least not made much difference.
What typically happens is the PC's have a run of bad luck and you think, "Gee, this combat is going bad. I should put on the kid gloves", and then the luck swings the other way and the PC's trounce over the encounter without further difficulty. Or else, the NPCs are having a run of bad luck and you really think it's going to harm the drama and the BBEG is going to go down without a fight, so you fudge, and then the luck swings back the other way and suddenly you've got multiple PC's down and your facing down player deaths or TPK if you don't now fudge your fudging.
Well, most of the time I change the encounter parameters or add new features (my usual way of improvising / fudging) not to determine the outcome, but to determine HOW we get to the outcome. So I do quite often add or remove monsters, add or ignore their powers, or add or ignore terrain and effects to make an encounter more fun. Sometimes fun is just not spending 10 more minutes defeating surprisingly tanky enemies, sometimes it's upping mooks' hitpoints because they were not up the standard of the main enemies, sometimes it's having enemies run away because everyone is yawning and it's time to wrap up. Sometimes it's keeping the enemy alive for a turn to let their nemesis have a chance at delivering the KO blow.
Many times it has been when playing with kids and not injuring their animal companions.
For me, the "typical" case is not "the PC's have a run of bad luck"; it's "I, the GM didn't get the encounter pre-planned quite right". Even though I am a good GM, I will make mistakes often enough, and admitting those mistakes and fixing them is, for me, the most common reason I modify and encounter in-flight. I'd estimate about 75% of the time it's to increase the difficulty, and I don't recall a time I've been unhappy with my decision.
It doesn't really improve the narrative
That's kind of an odd assertion -- I think the reason everyone does this is to try an improve the narrative. Earlier you said that it made your stories better 33% of the time. for me, it's virtually always made things better -- saved boring time, added fun elements, allowed a hero to meet their nemesis -- good times!
You really should be playing to find out what happens and not deciding what is going to happen because you like that story more. If you are frequently breaking the rules because you don't like the consequences, you should get different rules. It's better to question whether the rules really are providing the sort of game you want to have than to resort to illusionism.
I think you are focusing on
changing the outcome of a combat; the strawman here is the thought that people who adapt encounters mid-flow are railroading GMs who want their version of the story to occur, no matter what. But that's a rarity. Most of the time we don't want to modify it to enforce an outcome (I certainly never have done so and based on comments, others rarely do), we do it to
make the encounter itself more fun.
Even when we adjust the degree of challenge, it's not to enforce an outcome, but to make the challenge closer to how we had envisaged it. So when I "fudge" and encounter to make it harder, it's not because I want to change the outcome and have the players lose, it's to make the encounter closer to the fun that I was trying to design into it.
For me, this activity of ours is about having fun. That is the primary goal. Saying that you will not modify an encounter mid-flow to make the game more fun for people contradicts that goal. Now maybe when you do make these adjustments it doesn't work out for you, and so you don't do it. Fair enough, not everyone is good at all aspects of GMing. But if you
could modify encounters so that everyone has more fun, and you choose not to, then it's you deciding that your principle is more important than your players' fun. And for me, there are very few times that can be the case for any principle I use in GMing.