I’ve seen team bad guy lose quicker because they took 1-3 attacks to kill a familiar, when the PCs hit most of their attacks anyway.
Advantage just isn’t that big a deal.
Hell, I’ve got an owl familiar in one game that has mithral barding crafted by a gnome master crafter, so it rarely gets hit even when I have enemies try, but...so what? They spent gold and downtime and made a cool thing, even had it enchanted so it comes back with the owl and changes shape if they summon the familiar as something else. It’s cool.
The game very much runs just fine with the familiar not getting attacked.
My own character has a familiar that is a wolf, via DM permission. It’s fine.