Arab raiding effectively depopulated the north-Mediterranean coasts in some periods. But I think it was a lot more intensive than what the Vikings managed. And they were carrying off populations, which the Vikings did a fair bit, but medieval Christian Europeans didn't since slavery died out fairly early except in a few fringe locations like Scotland.The Black Death certainly killed more people and disrupted society more than any of the wars - possibly all of the wars combined, with the exceptions of one or two local campaign specifically aimed at the total eradication of a particular society (Charlemagne against the pagan Saxons, roughly where I'd start the "middle ages"; the failure of the Hussite crusades at the other end).
Something to consider is that warfare in the medieval period is characterised not by battles or sieges, but by raids. That doesn't always mean the raid is restricted to a region close to the border. The Great Chevauchee (led by John of Gaunt of Shakespeare fame) crossed France from Calais to the English king's territories in Gascony, looting and pillaging much of the way. Other armies covered equal or longer distances in their raids (in eastern Europe, Mongol successor states raided extensively, and did so for centuries). In other parts of the world raiding was more local, but where the borders were stable it could go on for long enough that the border regions basically developed farmer-warriors who lived in fortified villages and expected to engage in a certain amount of raiding or raid-defence every year. The impact on the population and wealth of eastern Europe was much greater from those centuries of persistent raiding for loot, slaves and random destruction than the effect on France of one great expedition. Yet persistence over time isn't necessarily going to do that much damage - the Viking raids lasted for a couple of centuries and England came out of them as one of Europe's wealthier kingdoms (Scotland didn't, but Scotland was poor before they began and didn't regress noticeably). Effectively you can say there's no "front line", and an army can appear anywhere; but that unless you're unlucky enough to be in that army's path you probably don't notice much effect; and if it comes by again next year and rides over you, then you're going to develop defences to make that harder and a society where most people are able to defend themselves and those that aren't have moved somewhere safer.