It sounds like you can, but with an unarmed attack. So unless there's some condition that prevents OAs (daze, stunned, etc.), you can always take an OA.
It makes me smile to think of a wizard headbutting a troll because he decided to walk away. :) But if I'm that wizard, I'll certainly take it.