Two ideas to help with that. Carry the familiar(and if it's an Imp it can be going invisible on it's turn for advantage) or Ready the attack action(presumably for a trigger like "the imp goes invisible") and just have the Familiar move up to the right enemy and trigger your reaction.
Not like...