If you want all your patrons to work that way, more power to you. I've had a warlock in the party with a patron of that kind. It was fine. I've also had a warlock with a patron that took an interest in him. He was one of several of her agents, but she gave him missions to do, and after he did...