Correct. Only in this case I don't (and wouldn't) have melee weapon use trigger AOOs, so not an issue for me.So every time fighter makes an attack Wizard can cast a spell at him as an AOO ..... and then I guess with your houserule no one can AOO that Wizard for his casting because it is an AOO ..... which brings us back to square 1, Wizard gets to cast a spell without anyone being able to interrupt her!
For counterspells and other interrupts, I do this to avoid M:tG-like counterspell wars. It works in Magic because in that game the stack goes last-in first-out, but in an RPG setting where in-game time is a thing and usually only goes in one direction, the "stack" has to go first-in first-out; meaning there's no way to cast a counterspell fast enough to counter another counterspell (Crawford's immensely stupid ruling notwithstanding).
I'd apply much the same logic to AOOs - by the time the second responder realizes there was an opportunity that opportunity has already passed because the first AOO has already been taken, and the chance is lost.