Think of it this way:
Drawing a weapon is just pulling it out of its scabbard, sheath, unhooking it, etc. In most cases, you would be familiar enough with where it is on your body to do this by touch, without looking down.
Now, putting a weapon away (as opposed to just dropping it) is a different story. To fit a dagger into a sheath, a sword into a scabbard, etc. requires lining up the blade with the slot in the scabbard. This requires a pretty precise set of movements in most cases and is extremely difficult to do quickly without looking down or breaking your concentration, hence it draws an AoO.
The examples I gave were for bladed weapons, but similar parallels can be drawn for other kinds of weapons (shoving a mace down through a leather loop on your belt, etc.)
I'm guessing this is what the designers were thinking. Makes sense, right?