In full disclosure, my training is in biochemistry, and not zoology.
I know that millions of years ago, Earth's atmosphere had a higher Oxygen content that allowed terrestrial arthropods to grow quite large.
Well, first off, this is a theory. Personally, I think that because arthropods were among the first creatures to go from sea to land, they exhibited a form of island gigantism, if you will. The theory is that a lack of predators removes any evolutionary pressure to be smaller; larger is usually better. With a lack of real predators for a long time, arthropods grew larger. Evolutionary pressure pushed them back smaller once predators developed.
Additionally, oxygen usually isn't a terribly limiting factor in larger creatures. Otherwise, you would have seen a global trend towards stuff getting smaller. Humans are the prefect example to the contrary, giraffes another.
The biggest issue with a larger size is two-fold. The first is nutrient uptake - the larger you are the more you generally need to eat. If the nutrients aren't there (competition, die-off, whatever) then a smaller size is more advantageous. The second is nutrient distribution - you need to develop a circulatory system that allows you to get nutrients (including oxygen) to the cells that need it, and spend energy doing it.
So while a higher oxygen content couldn't have hurt, I don't think it was responsible for increased size. Correlation is not causation, and all that.
I would hazard a guess that humans have had a significant impact on crustacean populations, at the least.
My question is this, though- given the size disparity between the Coconut Crab and the largest known tarantulas, is there some anatomical difference between the crab and the spiders that lets the former have a larger top size?
Realistically? Probably not. At this point it probably has boiled down to genetics. In fact, I'm not sure there's any data linking the progression of size of spiders in the current day - for all I know they could be getting bigger. Achan hiArusa brings up an interesting point with the lungs, so that might be something there... but I'm not convinced.