The timeline could be drastically and constantly altering multiple times a second as millions of time travellers swamp the entirety of history, but you'd never know it because at any given moment you only remember the current one. In half a microsecond, you'll be remembering an entirely different one because this one no longer exists. Or this one. Or this one.
Fritz Leiber worked this possibility to great effect in The Big Time and the related Changewar stories. Two basically incomprehensible powers are fighting for control of history everywhere and eveywhen. A very small fraction of people can remember changed history, and they get drafted into the war. Notably, the war is not converging on any kind of stable point.
Tom Sweterlisch takes up the “you can only travel to a future possible at the moment you go, and it collapses when you return” possibility in The Gone World, a really great story of sf horror. Explorers of the futures find a hostile presence far away and far from now, that gets closer in both time and space with every trip.
For sheer elegance, it’s hard to beat John Crowley’s novella “Great Work of Time”. Let’s see if I can explain the setup…
Envision a moment as a sheet of paper, or an infinite plane. It has its physical dimensions to the right and left. Down is all the history that led to now, and up is all the future implicit in the state of things st this moment. Time as we experience it passes perpendicularly to the moment. We don’t go up the plane; we go to a new plane, encompassing the next moment. And on and on, through however deep the stack of moments is.
A time traveler jumps off the train, so to speak. He
does move up and down the plane. Changes he makes to history change the moment, which the real (for all the rest of us) present moves further and further away in a directional orthogonal to the time traveler’s moment. Over (his) time, the traveler finds the moment weakening, its laws of nature and history becoming more malleable. Consequences ensue.