If we change semantics for clarity, and say the -4 is not because you are trying to avoid your friend, but it is because you are trying to be careful and pick your target. If you don't care (disliked/well protected Paladin or orc vs. goblin) you don't have a penalty, but have a risk. If you do care, and don't have Precise Shot, then you have a penalty, but no risk.
For being careful and missing by 4 or less, I personally think it's dumb that a relatively near miss would miss by five feet and unfailingly veer towards my ally and risk damaging him/her/it. I know... the swirl of combat and all, but I don't quite buy it. Cover is different, and I only enforce the -4 when the target is given cover by someone/thing the shooter doesn't wan't to harm or is trying to specifically avoid. Otherwise, the soft cover rules apply. Limits the coolness of Precise Shot, but... I don't feel bad.