My house rule is a bit more complex:
Look at the spell teleport. See that table at the bottom, that cross-references how familiar you are with a location to how on-target you are? Use the same table for polymorph, baleful polymorph, polymorph any object, and shapechange, except it's how familiar you are with the creature in question.
"Very Familiar" means you've seen the creature many, many times, and recently. Example: Cat
"Studied Carefully" means you've studied the creature in the past for the purposes of this spell, and/or made a Knowledge check. Example: Hawk
"Seen Casually" means you've seen quite it a few times, but not often and not long enough to really get everything perfect. Example: Crocodile
"Viewed Once" means what it says; you've basically seen it just once, or had some really good descriptions. Example: Dragon
"False Destination" means the player has never actually seen the creature. Example: Pit Fiend
And the outcome is likewise shifted. Let's say you were using baleful polymorph to turn a high-level target into a Dog.
"On Target" means you got exactly the form you were trying for. They're now a harmless dog, CR 1/3.
"Off Target" results in a similar form with CR 1-2 (1d2) higher/lower than intended. (Higher if it was baleful polymorph cast on an enemy, lower if it was polymorph cast on yourself or your friends.) They're now a Wolf, a Riding Dog, or a Celestial/Fiendish Dog. Feel free to advance the creatures an extra HD or two to balance it (but I use Soldarin's ECL system, where one HD only equals 0.4-0.8 levels, depending on HD type).
"Similar Form" (was Similar Area) results in a related form with CR 3-5 (2+1d3) higher/lower than intended. They're now a Hellhound, Blink Dog, Dire Wolf, Winter Wolf, or Shadow Mastiff.
"Mishap" results in a DRASTICALLY different form (although it should be something that could theoretically be linked to the requested form), with a CR 7-12 (6+1d6) higher/lower than intended (although it can't exceed what they had before the spell was cast.) They're now a Nessian Warhound or Retriever.
If the player was using polymorph in an attempt to get a better form, CRs go downwards for off-target changes; "Mishap" might result in an Ooze form or something relatively harmless.
If the player used shapechange, he rolled for this familiarity each time he tried out a new form during a single activation of the spell. Once he's successfully used a form during the spell's duration, he can switch to it without rolling again. But, once the spell expires and he casts it again, he has to roll again for each form he tries.
Also, with shapechange, if he got "Off Target" or "Similar Form", he couldn't change again for 1 minute. If he got "Mishap", he was stuck in that form until the end of the spell's duration AND couldn't dismiss the spell.
Wild Shape didn't use this houserule. It's assumed the Druid is completely familiar with every form he uses, and that the source of the magic WANTS it to work.
Anyway, the result of this change is that players were a bit more reluctant to use polymorph spells often. They still used them, but it only took one Mishap roll to make them a bit wary; even a "Similar Form" roll was pretty severe.
