Pushing things off cliffs is about the most fun you can have with your Athletics skill. During the Chapter 1 fight with the Quaggoths in Out of the Abyss, I adapted the OotA rules for defenestration (ruling: if you're pushed off a cliff but still within 5' reach of the edge, you get to make a DC 10 Dex save per OotA to catch the ledge; if you make it you are Prone but not falling; if you're pushed 10' or more off the edge you are out of range of the edge and so just fall to your death) and since the players fought the quaggoths on the edge of a cliff, there was a whole lot of pushing going on, on both sides. It was quite exciting for everyone but no PCs died.
Anyway, if your players are trying to defenestrate things, I would come up with balanced rules that prevent it from being a dominant tactic (saving throw as above; also remember geometry, and apply Shove-Aside rules per DMG if they are trying to push someone sideways instead of directly away) for either the players or the monsters. Other than that, I would encourage them to exploit the environment in exactly this way whenever it makes tactical sense. Even if shoving someone out of a window does only 1d6 damage, it's still a form of battlefield removal, therefore useful for defeating an enemy in detail. If the enemy isn't highly-motivated he may not even try to re-enter the battlespace; he may just lie there on the ground moaning. (It depends on the scenario and the goals which are bringing the PCs and NPCs/monsters into conflict.)