Here's my best guess:
Close Combat
To begin grappling, you must work through the following steps:
1) Grab the Opponent. As a standard attack, try to get a hand on the defender: an unarmed touch attack. This will often will provoke an attack of opportunity. If that AoO does damage, then the grapple attempt fails right here. You can make a grapple attempt in place of each attack you normally get.
2) Close In: Roll an opposed grapple check (base attack bonus + strength vs. base attack bonus plus either strength or dexterity). Small characters get –4, large get +4 and so on. If you lose, then that is the end of the attempt. If you win, then you move into the space with your opponent and are officially grappling with them. You also get to inflict the damage from a single unarmed attack on the opponent as you close in. This is usually d3+str subdual damage.
Fighting in Close
Once you are grappled (or grappling), then
· You can only use unarmed attacks or light weapons
· You have no threatened area.
· You lose your Dex bonus to AC against people you are not grappling.
· You cannot leave the space.
· Two weapon fighting is not possible.
· You can take no actions other than to attack, prepare an item (or spell component), or try a non-somatic spell or ability.
· Doing anything complicated while grappling requires a Concentration(20+) check.
On subsequent actions (after the grab and close) you can make an opposed grapple check as a standard attack option. You can do this once for each unarmed attack that you would normally get (The FAQ says, for monsters, to use the BAB to find the number of grapple checks allowed, and don’t add attacks based on number of appendages). For each one that you win, you can do one of the following:
1) Do unarmed damage (plus constriction damage).
2) Pin your opponent for 1 round (they are effectively paralyzed, but not helpless, until just before your grapple roll next round).
3) Break a pin that someone else has on a friend of yours in this same grapple.
If you are pinning someone, you can only do unarmed attacks against them. Other people can use weapons, and also get a +4 to hit the victim. You cannot attack, or try to pin someone else while pinning someone.
Breaking Close Combat
To get out of close combat you have to try to break the grapple as standard attack action. Roll either an opposed grapple check, or an opposed Escape Artist vs. the opponent(s) grapple check. You have to roll better than ALL opponents to get out of the pile.
If you succeed, then you can still move, or do a move equivalent if you have actions left. Technically, you are not prone when grappled and so you do not have to take the time to stand up before moving.
Attacks of Opportunity
The initial grab is an unarmed melee touch attack and so may well provoke an attack of opportunity from your opponent. If it does, and the attack of opportunity inflicts damage, then that ends the grapple attempt.
The “close in” move is basically a 5ft step and so rarely provokes attacks from nearby enemies. If you charge and grapple, or move and grapple, then you might provoke attacks of opportunity as normal (for crossing threatened areas). You have to be able to move far enough to enter the defender’s space in order to try a grapple. Note that the defender does NOT get an attack of opportunity because of this last five feet.
The unarmed damage for closing in occurs right before the move, and so before any attacks that the move may cause. It is rather odd that a 30ft move and grapple would be resolved as a 25ft move, a grab check, a grapple check, and then the last 5ft of movement; but that is the way it works.
Improved Grab
Certain monsters have this feat and it modifies the above rules as follows:
1) They can start a grapple as a free action after hitting (and doing damage) with a normal melee attack.
2) These creatures do not provoke an attack of opportunity for the initial grab, nor do they have to roll to hit for this grab. It already hit with the melee attack.
3) After the grab, the creature can do one of two things: conduct the grapple normally, or conduct a “improved hold”. Both options require a standard grapple check, though the creature suffers a –20 penalty in the case of an improved hold (I say “improved” because the PHB also uses the term “hold”.).
4) If the grapple is conducted normally, the creature does NOT inflict more damage when it moves in, since it already inflicted damage right before the free grab. But it does add “constrict” damage at this time if applicable.
5) On subsequent actions, more grapple checks can be made (up to one per attack, number of attacks based on BAB only). A success deals damage equal to the initial melee attack, plus any constriction damage that they creature may have. If the check fails, then the victim is still grappled, but takes no damage. These future grapple checks can result in special effects (like being swallowed) regardless of whether it is a standard check or an improved hold check.
A “improved” hold has the following characteristics:
1) Unless otherwise stated, this grab only works against opponents smaller than the attacker.
2) It uses one appendage to maintain the hold.
3) The victim is drawn into the creature’s space, though no attacks of opportunity arise from this.
4) The creature is not considered grappled itself: the creature does not move into the victim’s space, does not lose Dex, and can still conduct melee combat, and move, as normal.
5) Improved holds incur a –20 penalty on grapple checks.
6) Escaping from an improved hold is the same as escaping from a normal grapple, though the holder is at –20 on the check.