See (coincidentally), Morris et al, 1988 (ref below) in which the authors speculate that a wormhole could be turned into a time machine. Such ideas may never be physically realizable, but they do exits in theory. General relativity is most definitely not my field, and maybe ideas have changed since I last paid much attention to it - but ideas for time travel have been discussed in physics journals.
Morris, Michael S., Kip S. Thorne, and Ulvi Yurtsever, Wormholes, Time Machines, and the Weak Energy Condition, Physical Review Letters, 61, 1446, 1988.
Nobody disputes that physicists have been using the theory to discuss theoretical time travel for decades; it's your added "can only go as far back as when the time machine was built" part that we're disputing.