What's annoying about using BB? I've used it as a player and had a player use it when I DM, and...never seen any problems crop up from it in any context.
My general tactic is simply to use the swashbuckler class feature to hit and run on something that is primarily melee, so it has to either give chase or sit on it's hand, or use a less powerful ranged option. if possible, my rogue hides as a bonus action after getting that attack. In an open field, he'll often forgo the BB attack, and instead use his rapier and hand crossbow to double his chance of a sneak attack, or stay at range with the hand crossbow and avoid melee.
What i do find annoying, sometimes, as great a DM as my buddy is, is when we fight basically mindless stuff like zombies, and they avoid giving chase in order to avoid the secondary damage, or to avoid another character's damaging zone, or whatever.