The problem is, that's not actually how the broom of flying works. The broom of flying doesn't even require attunement, and there's nothing in its rules text to suggest that it particularly requires either use of your hands or your concentration, at least no more so than doing anything in combat...