I always liked the AD&D rule where you had to experience the thing before you could make an illusion of it.
If I had to throw together rules:
Illusions deal "real" damage. If later the victem of an attacking illusion disbelieves it, all the damage the illusion dealt to them becomes subdual. Illusions can't kill. Victems going to 0 or below through the attacks of an illusion simply become unconcious and all the damage becomes subdual. (Though some special ones can)
If an illusion attacks a victem in melee, it uses the caster's attack bonus. Dagger of Lath has a good idea here.
If the foe has actually experienced whatever it is that the illusion is mimicking, then he or she gets a +4 on his or her will save to disbelieve the illusion.
The caster has to have experienced whatever he or she is trying to duplicate with an illusion. Ever seen a sphere of annihilation? No? Then you can't silent image one. Want to major image one? Well, unless you have touched one, I doubt you get that right.
If you interact with an illusion using a sense it doesn't emulate (Like listening to see if the Ogre Silent Image makes loud stomps with it's feet) you get +4 to your saving throw when you try to disbelieve.
Foes subject to the special attacks of an illusion (Like a medusa's gaze) that the illusion can't duplicate the effects of, get a save as normal for the mimicked attack. (using the spell's DC, modified by any bonuses the subject may have such as familiarity with the effect) Failure means they believe the effect and fall victem to it. (So if they failed their save against an illusory medusa, they would hold still, thinking they were petrified)
Illusions inflicting effects that last longer than Instantaneous allow a saving throw each round to realize the ruse. (So after a while, the guy who was "petrified" by the medusa illusion would get tired of standing and realize "Wait a minute...") This doesn't negate the illusion, but provides the above +4 due to unemulated interaction.
Lastly: If you wish to disbelieve an illusion, you must do so completely; You must not resist any of it's attacks. If you do, then you believe there is an off chance it could be real, and the disbelief fails. Thusly, if the "illusion" attacks and you wish to disbelieve it, you must allow yourself to be struck. (Treated as helpless; You want to be struck and I am sure if it is a real creature, it will take that perfect shot) If it uses an effect that allows a saving throw, like a dragon's breath weapon, you must voluntarily fail the save. Improved Evasion cannot apply to this. Your defences cannot apply to this either; So if you have a ring of fire immunity / resistance, you would have to take it off to disbelieve the red dragon's breath.
As for assessing their effectiveness... it depends on how into it the player is. If they describe every little detail then I'm more likely to have the damage be real or full damage... if damage dealt is greater than the hitpoints of the creature then I suppose you could say they were scared to death (too much adrenaline and a weak heart)... I'm not a big fan of any save or die type spells, the phantasmal killer included, so I would probably allow it do deal damage based on the form it took. All of this is provided the victim fails his saving throw... keep in mind there are things to boost save DC's and if the player is extremely descriptive then I would add a circumstance bonus to the DC.
