So what you're saying is that it isn't unbalanced at all if the Patron is ignored as an element, it's a narrative element that can be used to further the Warlock character's story?
To me that says that the warlock isn't overpowered if they choose to ignore their Patron and the DM decides to not follow up on it, it's just that a possible story hook for the Warlock character is being ignored (possibly one that the player wants to have explored since they're going out of their way to offend their Patron). That's a very different thing in my mind than a claim that the Warlock class is out of balance if the DM isn't using their Patron to keep them in check.