Unless I am missing something, one solution to these various problems sounds fairly simple, but requires a house rule.
A B X
B is flanked by A and X. X is invisible.
Before X is perceived by B as attacking, A's attacks are not flanking (it's as if X did not yet exist).
After X is perceived by B as attacking, A's attacks are flanking.
X's attacks are always flanking and always invisible (until A walks away when they merely become invisible).
So, X gets +4 (both invis and flanking), all other flanking advantages (like Sneak Attack if able), and B gets no dex against him.
A gets +2 against B (flanking) and all other flanking advantages, but only after B perceives X attacking.
The round after B perceives X, B can ignore X (for example, if he does not consider him a threat) and concentrate on A. If B ignores X, then A does not get a flanking bonus. Note: A should get at least one attack against B with flanking (i.e. it takes a moment for B to decide to ignore X at which point he will defend against X) to make this fair for A.
However, if you ignore someone in combat, it should be the equivalent of being off balance. You are not actively defending against them. So, if B is ignoring X and if X was not invisible, then X should get +4 against B (flanking and ignored) and all other flanking advantages. If B is ignoring X and if X is invisible, then X should get +6 against B (invis, flanking, and ignored), all other flanking advantages, and B gets no dex against X.
There is no rules on being ignored, but I think the +2 to hit for both melee and ranged attacks equivalent of off balance works. Granted, actually being off balance also means that you lose your Dex bonus, but the problem is that many unfavorable conditions (like Surprise, hence, the reason to not use Surprise or Flatfooted here) means that the defender loses his Dex bonus (which makes it difficult for unrelated conditions to stack) and since Invisibility is one of them, I thought that ignored should not.
On top of this, until B perceives X's attacks, it is as if X is ignored (B doesn't know about him after all). So, until his attacks are noticed, X is not at +4 (both invis and flanking), all other flanking advantages, and B gets no dex against X(as mentioned originally above before I introduced the house rule), X is at +6 against B (invis, flanking, and ignored), all other flanking advantages, and B gets no dex against X until B perceives X's attacks.
So effectively, B gets to decide if he wants A at +2 with flank and X at +4 with flank and no dex, or if he wants A at +0 with no flank and X at +6 with flank and no dex. Either way, A and X are at the advantage.
This solves all of the problems mentioned (if I remember correctly), but it does introduce the +2 to hit concept of being ignored to do so. That's the way it is with unresolvable problems based on the given rules. You have to adjudicate a house rule to resolve them. Otherwise, some aspect of it will still remain unresolved (at least to the satisfaction of a given GM).
A B X
B is flanked by A and X. X is invisible.
Before X is perceived by B as attacking, A's attacks are not flanking (it's as if X did not yet exist).
After X is perceived by B as attacking, A's attacks are flanking.
X's attacks are always flanking and always invisible (until A walks away when they merely become invisible).
So, X gets +4 (both invis and flanking), all other flanking advantages (like Sneak Attack if able), and B gets no dex against him.
A gets +2 against B (flanking) and all other flanking advantages, but only after B perceives X attacking.
The round after B perceives X, B can ignore X (for example, if he does not consider him a threat) and concentrate on A. If B ignores X, then A does not get a flanking bonus. Note: A should get at least one attack against B with flanking (i.e. it takes a moment for B to decide to ignore X at which point he will defend against X) to make this fair for A.
However, if you ignore someone in combat, it should be the equivalent of being off balance. You are not actively defending against them. So, if B is ignoring X and if X was not invisible, then X should get +4 against B (flanking and ignored) and all other flanking advantages. If B is ignoring X and if X is invisible, then X should get +6 against B (invis, flanking, and ignored), all other flanking advantages, and B gets no dex against X.
There is no rules on being ignored, but I think the +2 to hit for both melee and ranged attacks equivalent of off balance works. Granted, actually being off balance also means that you lose your Dex bonus, but the problem is that many unfavorable conditions (like Surprise, hence, the reason to not use Surprise or Flatfooted here) means that the defender loses his Dex bonus (which makes it difficult for unrelated conditions to stack) and since Invisibility is one of them, I thought that ignored should not.
On top of this, until B perceives X's attacks, it is as if X is ignored (B doesn't know about him after all). So, until his attacks are noticed, X is not at +4 (both invis and flanking), all other flanking advantages, and B gets no dex against X(as mentioned originally above before I introduced the house rule), X is at +6 against B (invis, flanking, and ignored), all other flanking advantages, and B gets no dex against X until B perceives X's attacks.
So effectively, B gets to decide if he wants A at +2 with flank and X at +4 with flank and no dex, or if he wants A at +0 with no flank and X at +6 with flank and no dex. Either way, A and X are at the advantage.
This solves all of the problems mentioned (if I remember correctly), but it does introduce the +2 to hit concept of being ignored to do so. That's the way it is with unresolvable problems based on the given rules. You have to adjudicate a house rule to resolve them. Otherwise, some aspect of it will still remain unresolved (at least to the satisfaction of a given GM).