Here's a humorous reason why time travel may be possible, but we don't have any time traveler's.
They go back in time, but go back to the wrong location!
The Earth's movement is not static in space. It moves at a very fast speed. It moves thousands (if not faster, it's hard to calculate for me, but it's not just it's rotation, it's a movement around the Sun, then the Solar System's movement around the Galactic Center, then the Galaxy's movement through the Galactic Cluster and the Galactic Cluster's movement through space itself which may be up to millions of miles (Kilometers) per day/week/month/year).
Without knowing the size and mapping the universe, it may be impossible to know the exact location the Earth has been in the past (you would need a coordinate map for the Universe...for starters). So...the time traveler's go back in time, but as this would be breaking the time/space relationship to a degree, they go back to where the Earth is currently...except...there is no Earth in that location at that time period.