I can't imagine that the game would break if you house ruled that drawing/sheathing items is a free action, or something like that. It would make the Quick Draw feat pointless and potions would be a little easier to use, but I don't see what else it could seriously impact beyond believability.
One of my players wanted to be a marshal with three different firearms, so I wrote up a six-shooter, rifle, and double-barreled shotgun for him (they're based mechanically on the crossbows from the PHB/AV). He took the Quick Draw feat and loves planning out each round which gun he'll combine with which power, using the appropriate actions to draw/holster his guns and tracking what gun(s) he's holding. I admit, I don't really see the appeal, but at least for some players the juggling act is apparently enjoyable.
Edit: I just realized one possible abuse: someone uses a two-handed weapon to make their standard attack, then switches to sword and board at the end of their turn for the added defense. You may want to add a common sense "don't game the DM's benevolence" rule if you implement the above.