Yep, pretty much my view, though that is more of a Tavern Brawler type thing. I assumed that this discussion was for things like leaping off of buildings to add falling damage to attacks, or trying to trip someone into a fire place, or throwing a rug over someone to block their vision.
Just for fun, I'll share my ruling as a DM to a hypothetical player who proposes trying one of these things:
(1) Falling off buildings onto someone. If you're both about the same size, I'll take the normal falling damage, and split it between you and the guy you're falling on top of. A 40' fall onto him will do 2d6 to you and 2d6 to him--but he gets a Dex save (vs. your improvised weapon attack roll) to avoid the human cannonball. If
Rule of Yes applies, (i.e. first time anyone tries it) there will be no Dex save.
(2) Tripping someone into a fireplace: not that effective compared to weapons. Flaming oil only does 5 points of damage to whoever walks through it, so we can infer that 5 points of damage is actually a lot of damage (almost enough to kill a normal man). Fireplace fires are maybe slightly bigger, so I'll grant 2d6 damage; up to 3d8 damage for a large bonfire or 5d6 for a blast furnace.
You can shove someone into the fire using the normal shoving rules (or shove-aside DMG rules), so I'd allow a similar Athletics (vs. Athletics or Acrobatics) contest for a trip: it's functionally identical to a shove-aside, except that if the enemy is moving towards the fire at the instant you trip him, I'll let you avoid the usual shove-aside disadvantage OR have him land prone after the shove (your choice).
I'll let you trip anyone like that, whether there is fire present or not, and I'll let them trip you under the same conditions.
Rule of Yes is irrelevant here because I'm already giving you as much as you could reasonably expect, based on how Shoves work. It will continue to work just as well on second and subsequent attempts as on the first time you tried it.
(3) Throwing a rug to block someone's vision: if you just throw the rug, he can just bat it away, so you'll have to grapple him too and physically hold the rug over his head as a blindfold. You need two hands free, and you will spend two attacks (one to grab head using Athletics contest per grappling, one with an improvised weapon (rug) vs. Athletics/Acrobatics contest to force the rug over his head). If you don't have Extra Attack that may take you two rounds. As soon as you have two successes, the enemy is blinded for as long as you keep him grappled and the rug is intact. Rug may be attacked the same as a Net: AC 10, 6 HP for a medium-sized rug or 10 HP for a large one; immune to bludgeoning damage; any damage inflicted on the rug is shared equally between the rug and the person wrapped in it.
Similar blinding rules will apply to all manner of blindfolds, although different blindfolds will have different stats (HP/AC/etc.). You may acquire proficiency in Rug per usual rules for improvised weapons, using the Weapon Master feat or Tavern Brawler.
If
Rule of Yes is in force, it's probably just a single Athletics check to grapple and blind him.