Ever wielded a sword? Changing hands or shifting grip takes virtually no time. Nothing compared with drawing a weapon from its sheath and readying it.
And as somebody mentioned earlier, if you can open your spell pouch, look into it, scramble around a bit, curse because your cell phone, chewing...