A thing to keep in mind is that there's no reason to perform tasks on a Broom of Flying at high speeds. You can perform actions before or after you move, with the Broom remaining still, as it doesn't have a minimum speed to remain aloft.
I can understand not wanting to make using a Broom of Flying an easy task, given it's unlimited flight and superiority over other options. By default, however, it really doesn't seem like it's difficult to use, as it doesn't require any proficiency, unlike controlling a vehicle. You could, however, enforce a downtime activity to learn to ride the Broom (or allow one to "master" it with the same). Lacking this training, imposing disadvantage on attack rolls seems perfectly fair.
What's interesting is that no special checks or ability has ever been required to use the Broom. I went back and checked my AD&D DMG and and the only real difference is that it's weight limits and speed reduction were much harsher. It still offered limitless flight, when the Wings of Flying did not, though said item was certainly faster.
*As an aside, since I'd never had the reason to use a mount in 5e, I was surprised to learn there is no proficiency required to ride a horse, though Animal Handling does mention controlling a mount.
I'm not sure why no one has ever thought this item needs balancing before. Checking my 3.5 DMG, I was surprised to note the following facts:
The Winged Boots cost 16,000 gp and only allow the use of the Fly spell 3/day for 5 minutes.
The Broom costs 17,000 gp and allows for effectively permanent Overland Flight. It has more stringent creation requirements, but it only being 1000 gp more is quite nice.
The Wings of Flying offer unlimited 60' flight as per the Fly spell, but cost a whopping 54,000 gp, though arguably is easier to create than the Broom (it's caster level is set at 10th, instead of 9th, but the Broom requires Overland Flight and Permanency, instead of simply Fly).
I think it's safe to say that the 5e Broom's rarity should be upshifted to at least Rare, and possibly even Very Rare due to it not needing attunement. Though I suppose that brings up it's one true flaw, as I see it: once someone hears it's command word, anyone could attempt to take it from you, especially if you send it off somewhere (it would be amusing to see a duel between multiple individuals armed with that knowledge vying for control of the Broom, lol)!