I stick with the RAW.
In my group I've had several people that would abuse the system. Specifically it was first a Bard who wanted to switch from Dual Wielding, to Crossbow, to Casting. Then it was a Barbarian who wanted to swap between Dual Wield and 2H every other turn. Lately its a Cleric who wants to switch between Sword & Shield, Crossbow, and Casting. At this point all I have to do is ask, "so what are you dropping". It visibly frustrates them, but as a DM I feel if I relent then its a slippery slope.
Smaller incidents with pure casters have been when NPC's provoke AoO's and they want to attack. I ask with what, and invariably they say with their melee weapon. I then have to ask if they are holding their focus in the other hand or the lantern/torch. I usually just handwave it even though I know in their mind they are NOT picturing their Sorcerer/Wizard waltzing around with a dagger and a wand (Arcane Focus) at the ready.
What I do handwave away is the initial combat round. If the player says they have their sword and shield ready then I accept it. If they say they had both weapons drawn, I accept it. I haven't outright stated this, but my players have kind of picked up on it and don't question it...mainly because they know I would be all to happy to ask them what their standard "walking around" set up is. >: )