Total Defense
"You can simply defend yourself and move during a round as a standard action. You don't attack or perform any other activity other than moving your speed..." (PH, page 127)
Attacks of Opportunity
"You threaten the area into which you can make a melee attack, even when it is not your action." (PH, page 122)
You can make an attack of opportunity if you are able to make a melee attack into the threatened square - the important words being "into which you can make a melee attack". This can be interpreted two ways:
1) If you are physically able to make a melee attack, even if you choose not to, or...
2) If you are physically able to make a melee attack, AND you have not selected an action which prevents attacking.
I tend to go with the second option. Otherwise, what happens when a spellcaster casts a full-round spell, but has a weapon in hand when an opponent prompts an AoO? Would you rule that he still gets the AoO, even though he has chosen an action which normally forbids attacking until his next action? In both examples (total defense and full-round spell), the character has chosen an action that limits his options until his next action. This also neatly avoids the whole issue of what modifier to apply to the AoO, since Total Defense does not specify one, though Fighting Defensively does (-4 penalty).