Everything we know about how time travel might possibly work requires huge amounts of energy at the very least. That is a thing that is true.
Yes, but everything we know about how time travel might possibly work also involves materials that don't exist, and/or that we cannot manipulate, in configurations that should not be physically stable. Like, infinitely long, rapidly rotating cylinders of neutronium.
How we think it "might possibly work" is irrelevant, unless you posit that we get it by incrementally improving our ability to do it by one of the methods we currently think it might possibly work.
If instead you posit we get it though some fundamentally new science, then the old considerations go poof.
If we assume a fantastical easy time travel
We are assuming fantastical things with ANY time travel. There is no time travel that is not fantastical.