What's to stop them from doing it once they hit the next map hex?
That's exactly how it would go in my game because I resolve travel one hex at a time, so if there's uncertainty as to whether the party successfully travels in the desired direction and enters the adjoining hex that lies in that direction, a navigation check is used to resolve that. If the check fails, a different adjoining hex is entered. I always let the party successfully navigate a hex they've already explored, so retracing their steps would be trivial.
Unless every one of your terrain areas is deep valleys or dense forest, they'll be able to tell they're lost sooner or later, just from surrounding landmarks such as rivers, coastlines and mountains.
Yes, and that sort of thing is just fine. I'm just looking for some variation in outcomes.
Ultimately, if you're going to run a campaign in this style, you need buy-in from your players. If they're happy to wander off the beaten path, take the road less travelled, find themselves at destinations they never knew they were heading for, then they won't have their characters turn back the moment they know, out-of-character, that they're lost.
If they don't like those things, then you're making them play a scenario they don't enjoy.
This is true of all playstyles, so I'm not sure why you're singling out mine. Part of my style, however, is to minimize the flow of metagame information on the DM side of things. I'm not interested in policing my players' use of such information, and I want them to be able to inhabit their characters without such distractions. That's why currently when a navigation check is failed, not only is the party lost, but the party also
knows it's lost because that's information I'm giving out.
Usually retracing your steps would not really bring you to your goal.
No, but it would give you a sort of do-over, which I generally find undesirable. Your goal was to travel in a specific direction, which you could try to do from the new space, but you'd eventually have to correct for the mistake you made if you have a specific destination in mind. You may be right. I'd have to test it out in game-play to see if it works for my group.
Let's take your scenario of going through a forest and wanting to reach southeast end of it. After a failed roll you could narrate that they travel through the forest. You don't tell them where they are except they reach the end. My player would usually just say something like "Alright, we keep moving southeast until we reach the road."
Then I'd narrate something like "You travel southeast and eventually reach a river."
Then my players would start discussing: "Hmm strange, according to the map, we shouldn't reach a river." "Maybe we went the wrong way." "It could be the river over here look." "If that's the case, let's travel downstream, we should read the road then."
Already more fun.
(Note: My narration is simplified, I usually describe more.)
I'm a little confused by this example because it sounds like they don't know they're lost even though they've failed the check. They only notice they're lost when the landscape doesn't match up with their expectations. That's generally the sort of outcome I'm after, but I don't see how you've gotten there with a transparent navigation check.