The big issue is hard drive speed and the video ram on your card (my VRAM is 1 gig, which helps a lot).
Those running on cards with smaller VRam and lower system RAM, or who have non-Serial ATA hard drives are getting hit with longer load times. This is as a result of the incredible environmental texture detail in the game. At full eye candy on - it's a lot of 1024 and 512 dds textures that are loading on a per area basis. Add in a fair numbr of autosaves when exiting after a main story quest flag is set, and it adds up.
If your hard drive is thrashing and paging on load to deal with texture loading, you will need to turn down the texture detail to improve area load times.
Mine has been fine and those with similar high end systems seem to be doing fine. Those with mid range or low range systems are not so fine. It's not about how many FPS your GPU can push (though that always helps of course) - but load times are hard drive and vid memory issues, primarily.