Yeah, but that is forcing the players mechanical and story wise to stay on course. As a player, I wouldn't like that, to be forced to not stray away a little, not to have the opportunity to do anything else in the world. Such a game would feel like a dead world, where there is nothing outside the walls of the plot.
For playing D&D you need to create an Illusion, an Illusion of a living, existing world where you have agency in. With milestones and a fixed plot, you are killing this illusion. Your choices don't matter. You follow a preplanned path of encounters and if your dice a lucky, you succeed.
With having XP you are saying that there is the possibility to go off the rails, even if the players never take that opportunity, you say mechanically, the world is open to you. The illusion of an open world is stronger, the illusion that your choices matter is stronger.
With Milestones on the mechanical side, you take away that illusion. Your choices don't matter, the world is not open for you to explore.
The more railroady the story, the better XP help with the Illusion of choice. It will increase the game experience for the players and create a stronger illusion of a living world.