So it's "how dare you play this balanced or even underpowered class, better like your drubbings"?
Because this is now creating roleplay costs for not getting any benefit. Which seems to be exactly as bad as balancing great power by only having roleplay costs, just in the opposite direction.
As stated above in several places, I think it is very good to consider this stuff and make patrons part of the story, to include hooks and ideas and such. I draw the line at being punitive, and the problem is, the really really obvious and likely choice, especially nowadays due to BG3's Wyll, is punitive and harmful results foisted on one and only one player solely because they chose spicy flavor, creating massive harmful consequences merely because "warlock" was written on the sheet, while rarely to never doing this to anyone else. Not even divine characters, since those don't have the manipulation implied by the pact.
Let me put that another way. Do you ever threaten Warlocks with loss of power? Do you do so for light and transient reasons? If so, do you threaten clerics for similar reasons, e.g. divine whim may be adequate for them to lose their class abilities? If yes to all, then I wouldn't play in that game but I would at least respect its consistency. If no, you have exactly what I was talking about: punishing Warlock characters solely because of their class, not because of any desire to enrich the experience.