I might be in the minority, but I tend to look at D&D as a collaborative process. I come from a background in theatre, so I tend to use the rules of improv when I'm working on D&D. There are four answers to any question in improv; "Yes, and...", "Yes, but...", "No, and...", or "No, but...". Using that framework allows me to write a massive, sprawling world and campaign with a huge amount of flexibility for whatever the players throw at me.
While I as the DM hold a lot of the cards, my players are in on the story too, and I find that getting your players to buy into the world and get excited about it by elaborating on their character's backstory can be really beneficial. I don't mandate an in-depth backstory, but I do ask a lot of leading questions when my players are creating their characters: Why are they adventuring? What questions do they seek to answer? What is this character's goal? With only a few exceptions, players make the world better and more interesting when they go down a rabbit hole and write an amazingly elaborate backstory. I've had everything from a several-page biography to "I want to become the best gladiator in the world."
Do they occasionally get murderized by a goblin in the first battle? Sometimes. But them's the breaks. Same way that I can spend hours building an adventure, or a really cool monster, only for my players to hang a hard right and wander off into the sunset with Jeff the Throwaway NPC.
But that forces me to be creative. Why do my players love Jeff? How can I use Jeff to bring them back to main plotline? Is Jeff really as he appears, or is he eeeevil?
There are of course things I don't allow, or things I will reframe as goals, like the example
@CleverNickName gave of the warlock who wanted to be their own patron. That's the kind of thing I'd allow as an end goal that would take the PC out of the campaign and make them an NPC.
It can be really daunting to allow your players to put their grubby little dice-goblin hands all over your campaign baby (ask me how I know), but I've never found it to be an overall bad idea. It makes me a better DM, and gets my players to buy in to the world and get excited about the game.