Think of it this way:
Drawing a weapon is just pulling it out of its scabbard, sheath, unhooking it, etc. In most cases, you would be familiar enough with where it is on your body to do this by touch, without looking down.
Now, putting a weapon away (as opposed to just dropping it) is a...