I just consider the appropriate helm to be worn with the appropriate armor, and use the core rules (no special rules for helms).
But sure, I can suggest house rules:
Great helm (any helm with just a narrow viewing slit and breath holes)
Penalties: -10 to Listen, Spot, and Climb checks. -5 on Search checks. -4 on Sense Motive, Balance, Swim, and Jump checks. (these penalties do not stack with normal armor check penalties). You might also consider giving the wearer's targets the benefits of 20% concealment, unless the wearer is proficient in heavy armor.
Benefits: +10 circumstance bonus to AC vs. any attack that specifically relies on harming the head, such as a Mind Flayer's brain extraction. Big bonus to avoiding Gaze attacks. Immune to attacks from saps.
Closed face helm (any helm with significant face coverage, but less than a great helm's coverage; most barbutes, burgonets, greek corinthians, and open helms with face bars/"SCA helms")
Penalties: -4 on Listen, Spot, and Climb checks, -2 on Balance, Search, Jump, and Swim checks. (these penalties do not stack with normal armor check penalties).
Benefits: +6 circumstance bonus to AC vs. any attack that specifically relies on harming the head, such as a Mind Flayer's brain extraction. Small bonus to avoiding Gaze attacks. Immune to attacks from saps.
Open face helm (any helm that covers the crown and back of the head; some viking helms, roman centurian helms, open faced spangenhelms, any helm above with a visor in the raised position)
Penalties: -2 on Listen checks, -1 on Spot, Balance, Climb, Swim, and Jump checks. (these penalties do not stack with normal armor check penalties).
Benefits: +2 circumstance bonus to AC vs. any attack that specifically relies on harming the head, such as a Mind Flayer's brain extraction. +10 AC to attacks from saps.
Cap (any helm that covers the crown and little else; a Norman helm)
Penalties: -1 to Balance, Climb, Swim, Jump. (these penalties do not stack with normal armor check penalties).
Benefits: +2 AC to attacks from saps. Possibly a +1 circumstance bonus to AC for any attack specifically affecting the head and originating from above the wearer.
-z