That one goes both ways - if you aren't going to reach the walls, why do you care if they are there or not?
One of the main points is that the space of play is finite. That the space is curved such that you don't hit a literal wall does not change that there's only so much there, there. The fact that you don't hit a physical barrier created an illusion that the space isn't limited.
If having that illusion is super-important to you, so be it. But there's no real light between agreeing to have that illusion, and just agreeing that the space of play has some boundaries to a continent (like, say, the original Greyhawk) or a city.
Plus, the box is a metaphor. We are speaking about the physical extent of the imaginary world because that is conceptually easy, but geographic limitations are not the only ones we could consider. There are other expectations of play that can be considered agreed-upon bounds to your agency that don't suddenly make the thing into a railroad. As suggested previously - the agreement that you're playing an adventure game, and will be doing adventurous things, and not shifting into playing a farming simulation, for example, is technically a bound on agency.