XP are a deliberate metagame tool. They exist for the purpose of encouraging certain behavior and discouraging others. That is the whole point. Players doing whatever they feel like is not inherently desireable or good.
A good game is not all the things to all people. A good game has a focus. Rewards and incentives are a great tool to maintain focus in the campaign and staying within the campaign's premise. For one player it might be fun to play the game in whatever way seems most interesting at the moment, but when you have six people who are all interested in different things that doesn't really work. Also, in a campaign that is set up for a certain thing, the GM can prepare and create content accordingly. This becomes increasingly harder and less effective when the scope of the campaign is less defined. And when players can do anything, but nothing seems really pressing, it becomes hard to decide what to do to create new exciting situations.
A clear system that tells players "You'll always get rewarded for doing that thing" helps with all of that. It helps keeping thr campaign within its premise without having to set up invisible walls.
(That being said, D&D wants to be all things to all people and XP as a mechanic were copied over blindly out of tradition, without considering what they are supposed to incentivize.)