Note that this only applies to cover, which is entirely separate from the -4 penalty for shooting a target engaged in melee.
Here is the 3E rule, which I believe is pretty similar to the 3.5 optional variant.
Striking the Cover Instead of a Missed Target
If it ever becomes important to know whether the cover was actually struck by an incoming attack that misses the intended target, the DM should determine if the attack roll would have hit the protected target without the cover. If the attack roll falls within a range low enough to miss the target with cover but high enough to strike the target if there had been no cover, the object used for cover was struck. This can be particularly important to know in cases where a character uses another character as cover. In such a case, if the cover is struck and the attack roll exceeds the AC of the covering character, the covering character takes the damage intended for the target.
If the covering character has a Dexterity bonus to AC or a dodge bonus, and this bonus keeps the covering character from being hit, then the original target is hit instead. The covering character has dodged out of the way and didn't provide cover after all. A covering character can choose not to apply his Dexterity bonus to AC and/or his dodge bonus, if his intent is to try to take the damage in order to keep the covered character from being hit.
-Hyp.