You absolutely should stay!
Okay, now that the joke is out of the way, it really sounds like it's time to wrap it up amicably. You have strong real life reasons, I'd stick to those to keep everything good between you and your friend without burning any bridges. And you mentioned that you really like you character, but you've also said that you have little chance to develop him, or even cast Find Familiar.
It sounds like the playstyle of the other two players works with each other, it's not your duty to 'correct' something that isn't a problem between them. Get some sleep, and best of luck.