This argument has never made sense to me. It's not a thing I've ever seen as complicated or vague.
The wizard doesn't have an owl familiar, it has a fey/fiend/celestial that takes the form of an owl and uses owl stats & features. The warlock doesn't have an imp, it has a fiend that takes the form of an imp and uses imp stats & features (although arguably that's what an imp is.) In theory, you could have a fey imp, or even a celestial imp, as your warlock familiar.
Having an actual imp, a quasit, a sprite, or a pseudodragon as your familiar isn't a class feature, any more than having an apprentice, or a page, or a torchbearer, or a bodyguard is. Sometimes a PC will run into an NPC that is agreeable to some sort of partnership or service arrangement, and that's what having a real imp as a familiar is all about. You have to find the NPC, determine what the NPC wants in exchange for service, and provide that. You have to be nice to the NPC, or it will leave or betray you. The DM is ultimately in control of the NPC, even if you the player "run" it in combat.
A summoned familiar, by the spell or class feature, isn't an NPC. It is an extension of the character, and it is absolutely under the player's control just as the player's character that summoned it. The ability of an imp--a real imp--to serve as a spellcaster's familiar is not a function of the spellcaster's class features (other than, obviously, spellcasting); it is a function of the imp. The imp has an ability that lets it bind with a spellcaster and serve as its familiar, if the imp wants to and if its demands are met. If that changes, the imp (unlike a familiar summoned with the find familiar spell) can terminate the relationship.
To say that the Pact of the Chain makes it impossible for any other class to have an imp familiar is like saying that the find steed spell means that no player character without that spell can have a warhorse, and find greater steed means that no other class can ever ride a griffon. That's complete crap. If you want to ride a warhorse (not a summoned celestial in the form of a warhorse) then you go find a warhorse, buy it, and get on. You might need to learn to ride it, but if your DM makes you have riding proficiency (animal handling? tool: saddle?) then that's a proficiency that you can pick up with a feat, training in downtime, or by doing a favor for a riding trainer (see training as treasure.) If you want to ride a griffon, you go find a griffon (or a griffon egg) and raise it, train it, or charm it somehow. If you want an imp familiar, you go find an imp and make a deal with it.
The player's handbook is quite clear: the pact of the chain lets you have a fey spirit in the form of a pseudodragon, but it isn't a pseudodragon. A real pseudodragon is an NPC, not an extension of a PC by way of a class feature or spell. You aquire its services the same way you would any other NPC hireling/henchman/retainer, through compensation and persuasion. Has anyone ever seriously suggested that, unless you have the Noble background, you cannot possibly hire retainers? It just means you don't have retainers automatically--you have to spend your character's time and money to hook that up instead.
If you want a hobgoblin retainer, go find a hobgoblin and persuade it to serve you. If you do, it has a trait that lets it do extra damage to enemies within 5 feet of you. If want an imp retainer, go find an imp and persuade it to serve you. If you do, it has a trait that lets it serve as your familiar if you're a spellcaster. This is just playing the game, simple as that.
I don't understand how people think that they need a specific, detailed rule for everything in this game.