Basic rules, page 76
Knocking a Creature Out
Sometimes an attacker wants to incapacitate a foe, rather than deal a killing blow. When an attacker reduces a creature to 0 hit points with a melee attack, the attacker can knock the creature out. The attacker can make this choice the instant the damage is dealt. The creature falls unconscious and is stable.
Some spells (e.g. Shocking Grasp) have a range of "touch" and require you to make a melee spell attack.
Since a melee spell attack is not a melee attack, I would rule that you can't knock someone out with a spell. I don't see a problem with a house rule that states that a melee spell attack is effectively the same as a melee attack.