If you leave the AoO rules as is and add in this idea that making an AoO provokes an AoO, it changes the landscape without necessarily unbalancing things; Combat Ref would be a fantastic feat-even more so than usual. A typical character would think twice about taking an AoO however, and tactical movement would become slightly easier.
The consequences for special opponents are that mages and archers are at a slight additional disadvantage from the core rules (since they provoke a lot of AoOs but generally don't threaten an area and can't retaliate) and big creatures or anyone with with reach has an advantage (because they can whack enemies before they're in range to retaliate). On the whole, I think the retaliatory AoO idea is an interesting wrinkle, not necessary by any means, but not as unbalancing as the original changes.
The second change I made, which people are overlooking, is just what you can DO with an attack of opportunity (possiblydue o wording) I'm going to clarify the mechanic.
- It's a free standard action. This means that if an AoO comes up, the mage can cast a standard-action casting time spell. The Archer can make an AoO with the bow.
- I'd rule that if someone is making a ranged attack, it only provokes AoOs against someone with a melee weapon. (unless the Ranged attack is itself, an AoO).
- If an action provokes an attack of opportunity, only one creature will have the chance to take it. If multiple creatures try, the others will get in the way of the attacks (so the one with the highest init mod gets it; if tied, look at Dex Mod, if tied, look at Dex Score, If Tied, flip a goddamn coin).
I'm thinking a system where you can get more creatures doing AoOs based on size difference. Though the Wolf Packing Idea could work too. This way was my original concept though.
- Additionally, flat footed creatures, and creatures denied their dex bonus to AC for any reason cannot make AoOs.
-You can only make 1 AoO against a single creature in a round.
Tell me what you think of these restrictions. Maybe they're uncalled for, but we'll see.