I'm similar in that I also try to make the setting as character agnostic as possible, because the characters don't really matter. At first. However, since I don't start with a big campaign plot in mind, the character's backgrounds are vital; they completely drive where the campaign goes. In my home campaign, the party just resolved a huge chunk of the druid's background story, after almost a year's worth of games, and the current arc is now driven by the barbarian's, after some bounty hunters managed to lure him away from the party and kidnap him. So now they are focused on that. And the players themselves are adding details that I don't know about - for example, their next destination was decided because that's where the barbarian's player announced that his former mentor is currently hiding.
On the other hand, my spouse has given me very little background for their character, and is happy going along with the flow, though occasionally dropping little hints. That's cool, too! I love it that the story is unfolding for me as much as for them, and look forward to finding out what will happen in each game. For my preferred style of play, my job is to keep the world percolating away in the background, and the players decide how they want to interact with it. So character backgrounds supply many of the story beats.