Homebrew critters?
There's lots of options:
They can be individual tanks, but not particularly many of them at a time:
Spell resistance is good; pick a number such that the primary casters can get through it, just not easily. AC is good; again, pick a number such that the primary fighter can't dump too much into Power Attack if he expects to actually hit the thing (50% chance again - although you want this one lower, so that he can use Power Attack; set an AC about 6 higher than his attack bonus - he hits 75% of the time for his normal weapon swing, or 50% of the time power attacking for 5). Attack bonus is good; again, pick a number such that the primary tank gets hit - prefferably, about 50% of the time. Saves are good; pick a number such that the primary caster is going to be foiled by the save about 50% of the time. HP is good; pick a number such that, if the party is focusing on one, it will last X rounds (X should be about 2) on average.
They can be mostly fluff critters, but lots of them - and I mean lots - They go down in one hit from the primary fighter, and he cleaves through them easily, missing on maybe a roll of 2. They can scarcely touch the tank, hitting him on maybe a roll of 19. Casters go Zap, Zap, Zap to clean them up. Slight problem, though..... sure, they can kill 20/round.... but there's a few thousand, all attacking at once (they have something ranged, so they can keep attacking, although range increments and soft cover mean the vast majority miss even the low ac characters).
They can be sneaky critters - hiding in shadows, laying traps, running away from every straight-up encounter, zapping with different poisons from the shadows. Require the party have the ability to do things like walk on the cieling (tunnel with lots of bends, and the floor is coated in something caustic - set damage to suit; no primary caster? Climb checks, everyone.....). The ants handle this naturally as a matter of course (for that matter, make it so the floor isn't a trap per se, but simply a hazardous area - perhaps the grubs constantly secrete a digestive enzyme that quickly dissolves most organic matter so they can eat it by osmosis through their skin).
Perhaps they're clever bugs, and many of them have levels in Wizard (Enchanter) or Psion (telepath); most your players don't look like their Will saves are particularly grand; a little something so that the tank leaves a small amount of blood behind, a Wizard comes in after, and sends a little Demand to the party tank..... or Dominate Person from a closer range.
Perhaps there's a mix - Castes are standard in real ant nests - a zillion workers busily laying traps and re-routing tunnels to less invader-friendly areas, lots of recently hatched Warriors slaughtering themselves in a bloodbath up against the party, a few lucky harder-shelled Warriors with several extra years growth to up their potential; The Queen as a master of the mind, and a few Consorts (males) looking to find a sneaky way to impress the queen better for the next batch of grubs.
Edit: Perhaps there's a stage of grubhood wherein magic itself is part of their diet, and every so often there's an Antimagic Field because of it.
What flavor are you after? Is this nest an orginized thing, a forerunner of some kind of invading force? Is this nest a random thing, mindless insects following their programming to breed and spread themselves? Is it a mostly mindless hive with a few gifted overseers making the tactical decisions?
