Here's the simple version.
You start with full hit points, and lethal damage subtracts from them. So if you are a 100 hp fighter and you take 30 damage, you now have 70 hps remaining.
Nonlethal damage starts and 0 and goes up from there. So if you are that same fighter, and you take 25 points of nonlethal damage, you now still have 70 hps, but you also have 25 nonlethal damage.
Once your nonlethal damage equals or exceeds your current hps, you fall unconscious. So if that fighter takes another 20 hp, he's at 50 hps and 25 nonlethal, and is still conscious; but if he then takes 45 points of nonlethal damage, putting him at 50 hps/70 nonlethal damage, he falls unconscious. OTOH if he instead takes 45 points of lethal damage, he's now at 5 hps/25 nonlethal- and again, because his accumulated nonlethal damage exceeds his current hp, he passes out.
It's really pretty simple once you get the hang of it.
One thing does complicate the situation: Magical healing.
Magical healing restores
both lost hps and reduces your nonlethal damage. So let's say our fighter (who we'll say is at 5 hps and 25 nonlethal) has a cleric buddy who casts a healing spell that restores 15 points of damage to the fighter. The fighter is now at 20 hps and his nonlethal damage drops to 10. He's awake!
One really odd corner case, and one you'll barely ever see, is the combination of fast healing AND regeneration. Oh geez does this one get messy, but it's a great trick for 3e dms seeking a monster that's tough to keep down.
Let's say we have some templated, prestige classed, half-this and whatever monster with regeneration 5 (overcome by fire and acid) and fast healing 5, with 150 hps at max.
First off, anything but acid or fire becomes nonlethal damage. Secondly, every round, the critter's nonlethal damage drops by 10 (5 each for fast healing and regeneration 5) AND his lethal damage heals 5 due to the fast healing. Yikes!
Hope that helps!