I'm not entirely sure what you are looking for specifically, but monsters with a high AC (really high AC, as in first attacks of buffed-up warriors need a high roll to hit, anything beyond that needs a natural 20) tend to be pretty frustrating for warrior-types, while AC does not really hinder spellcasters.
We had something like this during the earlier levels of our current campaign, when we encountered two will-o'-the-wisps.
Also shadow dragons with their nasty capability to reduce the combat effectiveness of everyone pretty quickly thanks to their energy draining breath weapon, coupled with spellcasting, high AC and flight, tend to let warrior-types look pretty bad on the field of battle.
Then monsters with HUGE grapple checks and multiple arms/tentacles, that can basically grapple the whole group with the -20 penalty for not being considered grappling themselves, are - at higher levels - worse for warriors than for wizards, who can simply dimdoor out of the grapple and seek a safe spot to attack the creature from there. The warriors are left with inferior weapon attacks, which bounce off the creature's DR.
Then, of course, there are invisible, ethereal, etc creatures which spellcasters have better means to fight against.
And not to forget out-of-combat situations, where spells can often resolve situations, which otherwise would be huge obstacles.
Bye
Thanee