As long as i get to be the frog
Okay, and what empowers the DM to have any authority whatsoever?Only the social contract. The rules don't differentiate between whim and planned out changes. They just give the DM the authority to make any change. That's why, even if it's not interesting, you can't really take the social contract out of a discussion on restrictions on the DM.
Without the existence of the social contract, the DM could and would change things on a whim. There would literally be nothing stopping him and the players wouldn't leave the game if he did. The offense taken by players at playing in a game where the DM changes things on whims is rooted in the social contract and the expectations it provides.