A Iron Kingdoms dragon... in an open field... at night... with boiling-mud pit traps.
Very nearly a TPK, despite the party being 80% artillery (a rogue-paladin with UMD and a monstrous artifact cannon, dwarf and Rhullic snipers, a blaster mage and a monk with ranged ki attacks from a PrC). Absolutely hideous encounter, and the dragon only retreated due to consecutive crits. Its CR was about 6 above the party's, though, and we did slaughter it in its cave in the next session.
That was the scariest/most intense "boss battle" I've ever seen in an RPG, and well worth the trouble.
The iron lich (CR 22 + custom spells against oversized party, average level 13) was also a right good time, though mostly because of its crit immunity.
Annoyances? Anything with a miss chance (bloody SHADOWS were hellish against EL 14 PCs!), and great mobs of opponents about 4-6 CRs below the party level, just because of the time it takes to resolve their attacks. Also, most outsiders with their high and oddball DRs and their spell resistance.