This may seem odd, but I think it depends on the order of events.
Load, then fire is probably okay, because your last action in your round is to fire, at which point your buckler/shield is ready to defend you.
If you get stuck in a fire, then reload scenario then you get no defence from the buckler/shield, because the last thing you did was use your shield hand for something other than defense.
This may seem weird, but I think it fits within the rules well.
Alternately, you could simply disallow the use of the buckler/shield.