I think we are closer in position than it might seem.
Most of the changes I make are in-between campaigns. Most of those changes are talked over with my players and if a majority don't agree, the change typically doesn't happen. Sometimes if I feel strongly enough about it, it will happen anyway.
I agree with you that once the campaign begins, the DM is much more constrained with regard to changes. That constraint, though, is entirely from the social contract, not the rules. There have been rare times when during game play something breaks the campaign in some way, or some combo that I've never seen is so unbalancing that it becomes disruptive and I announce a change in the rules mid campaign. It's not a whim, because I don't change things on a whim, but it is a mid campaign unilateral change by me. I can do that because the rules not only don't stop me, but actually empower me with the authority to do so.