To begin with, swarm rules are complete crap.
A swarm's composed of so many tiny creatures that you can't affect them with weapons, but it has hit dice? Bull****. If it's composed of discrete creatures, it should work similar to something like Creeping Doom - a certain amount of damage (varying depending on the CR of the swarm) kills every swarm critter in the affected area automatically, affecting how much of the swarm is left and (if it was reasonably designed) how much damage it can do. The way it's written now, the swarm effectively has an infinite number of creatures, until it looses all HP, when they all die.
Or, if you're going to give something Hit Dice, do away with the "there are so many you can't possibly kill all" nonsense, and just give it an AC with appropriate size and Dex modifiers - those rules are there for a reason...
And the golem swarm is just a really bad application of a crappy template... If you don't have the right spells (And even then - oh, 5d6 damage to a 12HD creature... Yeah, that's a lot of help

), you're stuck trying to use alchemist's fire, acid and torches against it (provided it doesn't have energy resistances - can't remember now.)