If directions are just eye candy to you that's fine, especially when there is no meaning to the map anyway. It not a way that I enjoy playing as much. And sometime players have to make sub-optimal choices with a lack of information. this could be because they didn't plan, research, get organised, or it might be that there is too much time pressure and they just need to take a choice. If there is no difference between the paths (like in your game) then it doesn't matter. If the choice will have an effect (East they will go through the swamps, but if they go north they will pass close to a necromancers domain), then yes the choice can matter.
The choice
mattering is not synonymous with
player agency, though, which was the notion I was engaging with.
Simple example: suppose the GM has decided that, in land X of the campaign world, holding out one's hand out to another upon meeting them is a grave insult. And suppose that the players (i) do not know, and (ii) inadvertently teleport the PCs to land X (say via teleport mishap, or a Well of Many Worlds, or whatever). The GM tells the players that the PCs appear not far from some NPCs. One of the players then says of his PC, "I walk up to the NPCs, arm extended as if to shake their hands in greeting".
Now the player's choice here matters to the outcome - the GM describes the NPCs as grossly offended and commencing to attack the ill-mannered outlanders - but it was not any sort of exercise of
agency by the player (was it?). The relationship between the action declaration and the outcome is mere dumb luck. Like the players choosing to have the PCs go east (and therefore encounter trolls and hydras in the swamp) rather than north (where they will meet the necromancer). If the players don't know these geographical facts, then it's just dumb luck.
In a Gygaxian dungeon, there are a range of resources available to the players to eliminate dumb luck in this sense - divination magic in particular. But once you get to campaign worlds on the scale of some of those you mention (eg FR), then divination magic ceases to be very relevant. It's just luck.
Also: why do you say that in my game direction doesn't matter? If the players (in character) want to find the pyramid the orcs were heading towards, they have to head further east into the Bright Desert. If they want to recuperate in the ruined tower (which they did) then they have to head north to the foothills of the Abor-Alz.
But these choices were not made blind. The players chose to prioritise recupration over exploration, and therefore headed north.
In contrast: when the PCs were drifting through the ocean, hoping to be rescued, we didn't worry about which direction they were drifting in, as there was nothing that turned on them trying to drift one way rather than another.