If the small encounter doesn't lead to either meaningful loot or facilitate access to the milestone, a lot of players will regard it as a waste of time. And the thing about 5e is that you really can't hand out too much loot, so you can't reward too much meandering and dithering. In a typical 15-level adventure, you can offer maybe 6 or 8 extra side quests with interesting magic items. Without XP, the only reward for fighting a random purple worm in OotA if you're doing milestone leveling is "you didn't die." There is absolutely no good reason to go worm-hunting if you're a bit underleveled. The risk is death. The reward is nothing. Thus, the tactically correct decision in any wandering monster encounter is to flee, since fighting wastes precious twice-a-month play time and gives you nothing in return. Players can and do figure this out.
The different incentive structure of XP-based leveling facilitates design where the players engage on their terms, and decide for themselves whether the reward for poking about in this cave or slaying this monster is worth both the time and risk, and at what point they feel confident to spelunk deeper in the dungeon or push further into the frontier. Having DMed both with the same players, there is a world of difference in how they behave in the context of different incentive structures.