The swarm was bad. It came closest to killing us of everything in the Whispering Cairn. I might suggest swapping it out for another monster.
In fact, that's a tactic you would be wise to keep in mind throughout Age of Worms. Sure the big bosses/fights are all specially statted and junk, but...