We have periscope hand crossbows now? That's new.

Even if you could have just the "top" of the crossbow sticking out, I have no idea how you would aim.
D&D doesn't really map getting your hand cut off, so I would rule that the enemies are allowed to attack which could include a grapple followed by pulling the owner of the hand out of the hut. Many attacks/spells would work just fine even if only the hand is exposed.
I'd probably just rule that it doesn't work because of the simplified nature of the rules don't cover it.