Knocking somebody unconscious was one of the first house rules I put together (as it was in many of the earlier editions). I want to encourage not murdering everybody, and I also want to be able to model the types of scenes from books, movies, TV shows, etc.
However, this has to be countered by the fact that being able to bypass hit points entirely and taking somebody out of the fight must be addressed with care. Particularly because the possible attempts will be far greater against the PCs than for them.
So here's how we've handled it (after some modifications) and it seems to be working well.
First, being unconscious has nothing to do with how many hit points you have.
Unconscious
While unconscious, you make a death save each round. Three successes means you regain consciousness. Three failures before three successes simply resets the count.
Sap
I put together some rules for special maneuvers, such as a sap. You have disadvantage on the attack, and the target gets a saving throw DC 8 + your attack modifier) or you impose one of several conditions, depending on the type of attack. None of the possible conditions totally removes them from the fight.
In the case of sap you incapacitate the target until the end of your next turn if they fail their save, unless your attack is a critical hit. In which case they are knocked unconscious if they fail their save.
Critical Hit
The other opportunity for knocking a creature unconscious is a critical hit with a bludgeoning weapon. In our house rule, a critical hit does double damage, and the target must make a Constitution save (same DC). If they fail the save by more than 5 points, they are knocked unconscious.
We feel this is more realistic, more interesting in a story sense, and not likely to happen so frequently that it becomes a problem. Even at higher levels, the saving throw DC is not so high as to be unreasonable.
In the case of the assassin, their advantage to attack is countered by the disadvantage of the special maneuver, so they would just have a normal chance to hit. That still gives them a better chance than other characters, but not so much that it's unbalancing.
Incidentally, the other conditions that we allow for a special maneuver are blinded, prone, restrained, or suffocating (1 + CON bonus/rounds, minimum 1 until unconscious. So I guess that's the other option). Suffocation can be caused by garroting somebody, DEX save to avoid, Grapple to escape in subsequent rounds.
Ilbranteloth