My suggestion for flanking:
The attacker chooses one hex he occupies, and one hex an ally occupies. Both hexes must be adjacent to the defender.
If every possible line drawn between the two chosen hexes passes through a hex occupied by the defender, then the attacker is flanking with the chosen ally.
What this means in practice: The attacker chooses the most advantageous hexes possible to determine flanking. If a hex the defender occupies is completely interposed between those two hexes, then the attacker has flanking. If there's even one that can be drawn between the chosen hexes which doesn't pass through the defender's space, though, there is no flanking bonus.