Yes, time of day makes a huge difference.
If it is around lunch time, I'll load a page, go off and read the NY Times for a bit and come back to see if the page has loaded properly. Early in the morning, it is downright snappy.
Now, the question is why? Is the load that high around lunch or...