The PHB has a glossary entry for "damage" that isn't particularly helpful here, but the real question is: does it make any sense that a Swarm of X would be immune to damage from a Swarm of Y. (And the answer is: No, not really.) Swarms are immune to "weapon damage" because you can't hurt 1,000...