The spell you were originally looking for is apparently
Spectral Farce not
Phantasmal Farce. I remembered it as Phantasmal also from a Dragon magazine, but found it thru the excellent
Dragon Dex spell index with Ctrl-F "farce".
Anywho, it's in issue 229 which is in the awesome CD collection from ancient times so I have an electronic version for easy copying. Here you go.
Spectral Farce (Alteration, Illusion)
Level: 3
Range: 60 yds. + 1 yd. per level
Components: V, S
Duration: Special
Casting Time: 3
Area of Effect: 40’ cube + 10’ cube per level
Saving Throw: Special
The spectral farce spell is an illusion that makes images appear less believable, whether the images are real or illusory. If the images are real, the spell gives onlookers a reason to believe that they are fake. For example, spectral farce could make a charging red dragon’s movements seem unrealistic, and also silence the sound of its claws raking against stone as it landed in a cavern. This could make enemies of the dragon believe that it's an illusion and take actions to disbelieve its existence.
If the images are illusory, the spectral farce spell can be highly effective at promoting disbelief. It grants a +4 saving throw when determining whether a viewer believes an illusion is real. If the viewer has already failed a saving throw, there is a chance that another saving throw will be allowed after spectral farce is cast. The chance is equal to 50% plus the level of the spellcaster.
This spell can have all of the elements utilized in a spectral force spell, including sight, sound, smell and heat. Sometimes these are used primarily to mask an element, such as the case above where a dragon’s claws were silenced.
Definitely the kind of spell for the tricks you're looking for. But the same effect could be achieved with the creative use of existing illusions to give a cartoonish, surreal or phantasmal look to real objects or creatures. Or you could simply establish your enemy caster as a summoner first, and then mix up the summons with illusions of the same things. Since your party will already be used to monsters popping out of nowhere to attack them they'll have no justification to attempt disbelieving illusions that do the same thing. And if they DO start disbelieving anyway, that's when you throw in real monsters that deliberately look false via illusions. So now your real "false" monsters get automatic surprise attacks!
make sure your caster maxes out sleight of hand so he can use deceptive spellcasting to keep your party from using spellcraft to know what he's actually casting.
Also take the Deceptive and Invisible Spell feats from Cityscape.
Lightning bolts that come from nowhere will have them jumping at shadows and attacking "invisible" casters. And an invisible
fireball will just completely screw with their heads!

Would an invisible
obscuring mist make monsters look partially transparent? Looks like "illusionary" monsters.
