Many SCA fighters can shuck a shield in 2 seconds. Less than a full action, I'd say, but not much. Stowing it on the back takes much longer.
Fencers can usually ditch a buckler safely out of the combat in the same time. Restrapping it to it's buckle on the belt takes both hands and about 5 seconds.
And this could take even more time in the chaos of a real battle where your life is threatened.
Bucklers are very different from shields in that they are usually gripped by the fist, not strapped to your forearm.