This scenario sounds like the rogue is attempting to sneak up on someone and knock them out? I would argue that until a fight breaks out the combat rules don't apply, so sneak attack damage/weapons/finesse etc are moot until that point.
One thing the rules don't describe is a non-combat knockout scenario. To me asking for the attacker to reduce the target to zero HP is ridiculous because the target is not fighting back (which is what HP models). So to knock an unsuspecting target out I'd say it's simply a Str vs Con contest. The Rogue (assuming they successfully sneaked into position) wallops the target with the cosh, bam. Roll Str + ability bonus + bonus for the heft of the cosh (+2, +3?) compare to the target's constitution. If it beats the target the target is knocked unconscious (and takes the bludgeoning damage) or if it fails, target takes damage and a fight probably breaks out.
Attempting to knock out a creature one size larger (or same size but wearing a helmet) is done at disadvantage. 2 sizes larger and it's not possible. Smaller creature is done with advantage.
If I've misread the scenario my apologies, but I posted this in case it's relevant (or at least it's useful to others

)