I would rule against it - the player is trying to eat his cake and have it too. It also goes against the intent of the invocation, IMO. Plus, allowing it encourages players to shop around for other advantages they can get by swapping around pacts and keeping some of the advantages of the swapped pacts. Do you really want to open that can of worms?
To me, this just seems like a strategy to get a familiar for free before taking the pact they always intended. Had the player started the character at level 5 would you still let them have the familiar if they told you they had swapped pacts as part of their backstory?
Finally, I think the analogy to using a scroll is nonsense.
Edited to replace the word subclass with pact - was tired