Again, because there is no such a thing as absolute coordinates in space. Outside a planet there is not even something as simple as up or down.
The only way to make sense of space is through frames of reference, and while these can be arbitrary, many aren't and are tied to natural phenomena, like say, Earth's gravity.
There's little reason time travel would magically erase that. It would have to be tied to a frame of reference. Be it the entrance to a wormhole and then fly back to the solar system, or something even more fantastical like an energy portal or "altering the subatomic particles that carry time within matter". Just like you can't randomly end up in the magnetosphere just by walking.
How would you know, we haven't even seen the entirety of the Universe, much less even mapped out the Galaxy (or even mapped out this sector of the Galaxy)?
That said...
I never said absolute coordinates, I said coordinates.
Mapping out the universe with Coordinates is exactly the same as mapping out coordinates on the Earth or the Solar System. The same rules and ideas apply. Just because one is smaller, and one is bigger, does not change the rules of physics or how they are applied. They may be applied on a bigger scale, but they still can be applied.
You are correct, coordinates rely on the relation of spacial objects and our measurements and connections between them. This is how we came up with some of our coordinates.
223 Hillside Road, Manchester, UK is a coordinate of sorts. It enables one to figure out where a location is. You may not know how to find it as it's not a lat/long coordinate, but not all coordinates are specifically as you seem to want to allude to.
Some may be...it's 10 Kilometers from the red farmhouse that you turn left at after you travel 60 Kilometers down the road which is more what you would be saying is impossible as that is talking about a coordinate or location in referencing the location of another object.
However you do it, without a specific way to find the location of things in the past and where they were exactly, it would be impossible to travel to that point in time of their location.
If you traveled back 100 days to the exact location where the Earth is today, you would be in space. You would not find yourself on the Earth. In order to travel to where you want to on the Earth 100 days ago, you would have to first travel to the location where the Earth was at that point, and then move back to the time you wanted to be at.
This is why, if you want to go back to the Earth or a specific point on the Earth in the past, you will need to travel through Space and Time (like a Tardis and a Timelord do) rather than just traveling through time itself.