There is little or nothing the PC can do to protect themselves from an invisible attacker. They have no way of knowing where the attack is coming from before it lands, therefore the invisible attack gets advantage. In the meantime, if they ignore the other guy on their opposite side that enemy will get advantage because once again they don't see the attack coming.
Two attackers and not knowing where the next attack is coming from is worse than paying attention to the one you can see. The invisible attacker already gets advantage and by the rules advantage does not stack. It's not that the person being attacked is just giving up and standing still, they're just paying attention to the attacker they can do something about. In that case the invisible attacker still gets advantage but the other opponent doesn't gain any advantage from flanking.
I stick by my ruling and my logic.