I'll second the Rhyme ofthe Ancient Mariner. When I was younger, I memorized most of it. That last couple sections were too dry for repeated reading, but the first half still keeps me coming back.
I've also got a collection of Ovid's writing. Much of it is very inspireing.
From "Invocation:"
Third came the age of Bronze, less soft than silver. And men in bronze were quick with sword and spear, yet all feared Jove. Then came the age of Iron and from it poured the very blood of evil...
...not only crops were hoarded; men invaded the entrails of the Earth down deeper than the river, where Death's shades weave in darkness underground; where hidden from the sight of men, Joves treasures were locked in night. There, in his sacred mines, all that drives men to avarice and murder shone in the dark: the loot was dragged to light and War, inspired by the curses of Iron and Gold, lifted blood-clotted hands and marched the earth.
Not bad for the year 8 A.D., eh??