I would argue that protection from evil does not work against a harpy (though that is secondary to the discussion at hand). The spell you have in mind is probably silence, since the harpy song is a sonic effect.
Won't it boil down to the same thing? Cleric casts silence (or casts it from a scroll), entire party squeezes inside, then find a way to slay the harpy, chase it away or bypass it. At the end of the day, it is still resources expended, just that the majority is spent on defensive buffs allowing you to resist the harpy's song, rather than actually defeating them (because note that the harpies can still fly, and possibly wield ranged weaponry). Even if you had cast invisibility on everyone to sneak past them, it would still be resources expended.
So does the encounter go from being too threatening to a tedious exercise in dice-rolling? Or can it still be kept interesting?
Yesterday I GMed a high-level assault upon a meteor that had broken away from the prison plane entrapping Tharizdun and landed in the bottom of the ocean. Tharizdun was still trapped inside (and thus at half strength) but was building up an army in the surrounding waters. (The adventure is in RM, not D&D, and mixes bits and pieces of Beyond Countless Doorways, When the Sky Falls and the online WoTC Elder Evil adventure converted to RM with RM stats for dark gods and prison planes from ICE's C&T II and The Curse of Kabis.)
Most of the execution of the encounter was in the planning, which involved the party going in under extended Time Stop, bringing Tharizdun into their timezone and knocking him unconcsious before he could act, then using Force Shape Change to turn him from Voidal Energy form to human form so that he could be carried unconscious back to the dead star where he had been imprisoned, ready for that star to be reignited.
As it happened, once the party had prepared its assault, the encounter posed comparatively little challenge, though there were a few minor hiccups both going in and getting out. But the party was never quite sure that it would work - in particular, it was always possible that Tharizdun had already graduated to full strength, and thus would be much harder to knock unconscious in a single surprise round than anticipated. Thus, the encounter was not boring. In fact, it is one aspect of the climax of a 10-year campaign.
But if every encounter is to involve this degree of planning and suspense in order to make the use of preparatory, buffing and protective magic not just seem like a tedious fun-killer, then I don't want to have to GM 10 encounters per level. One like that is enough for me, in terms of the time it takes for me to plan it, and for the players to play it out (including the planning stages, which for my group - who can be slightly excessive about these things - itself took multiple sessions, including finding suitable NPCs to help them buff for the mission).
Because D&D presupposes a large number of encounters per level, it is incumbent upon the game to make it easy for a GM to keep them interesting, rather than being either cakewalks or TPKs, without each encounter requiring an inordinate investment of player or GM time.
For me, the decision as to whether or not I need to take Silence in order to save the party from a random encounter with Harpies falls on the tedious side of the line, not the suspense-building side. Others' mileage may vary.
rolling a natural 1 on your save is a metagame concept. In-game, your character won't know that he will fail a particular save only on a natural 1. When he does fail his will save against said spell, his reaction should be more along the lines of "darn, I let down my guard for just 1 moment, and this happens" or "I just wasn't strong enough", rather than "crap, I must have rolled a 1. Just my luck".
He wouldn't know why he failed the save, just that he did, and this would influence his future responses accordingly
So you're saying that the character should think "I need to do exercises to strengthen my self-discipline", which mechanically are reflected by taking the Iron Will feat? Is the player, who takes the view that a Cleric with a high will save has nothing to gain from Iron Will (because that won't stop any 1s from being rolled) and so chooses to take a different feat next level, an evil metagamer?
Personally, I prefer a system in which an autofail is not a metagame concept (eg fumbles in many systems, or narrating a natural 1 on a D&D attack as "No matter how well you manoeuvre, you just can't find an opening in your foes defence." This doesn't then create a dissonance between game and metagame which gets in the way of rational character building.
