They're annoying but doesn't the DM have the choice of what appears?
For conjure animals, the relevant text is:
Choose one of the following options for what appears:

One beast of challenge rating 2 or lower

Two beasts of challenge rating 1 or lower

Four beasts of challenge rating 1/2 or lower

Eight beasts of challenge rating 1/4 or lower
(with similar language for conjure woodland beings and conjure minor elementals; summon lesser demons has the dice pick the option).
I am interpreting that as the player choses which option and the DM choses the critters (with an implicit notion that the critters are the maximum CR of the option unless the player requests something weaker). The DM can select out some of the more annoying monsters, but I honestly don't find any of the monsters that annoying by themselves. For me, it isn't the specific critters, it is 8 new critters on the field (I haven't actually had anyone upcast any of these spells yet), of course I have circumvented it by bribing the players (you can get 1 CR 3 monster of the appropriate type instead of a CR 2 if they use the spell at the base level). I like any given PC being able to summon a flunky monster (even if they have to use a magic item to do so), but I find anything that lets them get more than one monster frustrating.
One idea I saw recently on Enworld might get me to change that, where the druid assigns a conjured critter to every PC and let's the PC's player run the critter. The opportunity to try that out hasn't happened yet, but there could be some extra special loot for the druid if it does.....