In most fantasy I have watched or read there was never a character in the fiction which followed around and mapped everywhere the group went. Because it is something no one needs in real life. You can go in to unexplored areas, wander around and leave without mapping a thing and still being able to know where everything is. You can even describe it to someone who has never been there and they could follow the same path based on just a verbal description.
I would suggest that was because traditionally it was used in areas that were, to one degree or another, both extensive and labyrinthine. It is entirely possible to get lost both leaving and trying to find your way back to specific places under those circumstances; I've seen people regularly do so with extensive suburban housing complexes on numerous occasions. I think you're giving people credit for more consistent spatial memory than is commonly the case, especially under the circumstances at hand (similarly, I've seen people get lost in extensive buildings, especially when the signage was inadequate.
Essentially, if your premise was true, most maps wouldn't exist in the first place.