I did not care for the swarm rules in 4e. (And to be honest, I don't remember what they were like in previous editions.)
The idea that you can deal any kind of damage at all to a huge number of minuscule creatures with a sword or axe is absurd. Swarms should be way scarier than that.
I think the problem is in treating them as monsters. A swarm should be treated as a trap or obstacle, like an out-of-control fire. It can damage you if you get caught in it and you can't stop it with weapons, but certain spells might help. Without magic, you just have to be creative with the environment to have any hope of containing it.