The way I interpret the Moving a Grappled Creature rules, the attacker can drag the creature to the cliff, but it takes a shove to push them off since Shoving a Creature is the rule that resolves pushing someone away from you. So that's two separate attacks to make it happen. A lot of DMs just say they're dragged off the cliff by grapple-drag-release, but I don't rule it that way. For someone with Extra Attack, they can still get it done on their turn if they succeed at both checks. For anyone else, you have to set it up on one turn and shove the next.