The way I see it it is this: The first level PCs arrive in the town at the edge of the wilds. There's an old mill infested with rats down by the river, a purportedly haunted mansion on top of the hill said to still hold the family's secret treasure vault, and an ancient border keep a few miles away where hobgoblins have made a forward base, probably with nefarious goals.
What tracking XP does that I don't think milestone leveling can do is allow the players to decide amongst themselves which of those things they want to take on, weighing for themselves the risk versus reward involved (with a significant chunk of that reward being XP). Tracking XP in a granular way also lets PCs probe any of these and only push it as far as they want to before pulling back. They are rewarded for what they actually choose to do (if they survive), not based on how many sessions they play or other arbitrary goals.