I was responding specifically to your "There is nothing they can do about the invisible enemy one way or another, so why even bother defending from an attack you cannot see coming? " use case, with the assumption that the defender was choosing to ignore the invisible attacker.
If a defender took a deliberate action to not try to defend from the invisible attacker to focus entirely on the visible attacker, I would rule that the visible attacker no longer benefits from flanking, but the invisible attacker will auto-hit, and possibly auto-crit.
As you said, "The rules don't cover this and as a DM I don't follow the letter of the rules when they don't apply", so I think a particular ruling is needed to cover this abnormal use case.