I'd guess there are two primary instincts that guide it:
1) D&D was created by Americans. It reflects a lot of frontier tropes, and exaggerates them even further.
2) If you have a bloody huge setting, it will take longer for you to get to the point that there are no more orcs to fight.
I suppose. It's just so hard to wrap my head around. When travel times between population centers becomes months, it makes trade virtually impossible. ((Yes, yes, Silk Road and all that, but, on the Silk Road, for the most part, you had population centers spaced at worst a couple of weeks apart.)) Something I've noticed with a lot of published settings is that the setting designers weren't coming at it from an economic standpoint, but, rather, going with what looks cool.
Kinda like what S'mon is saying.
Depends on the setting. Ansalon (from DL) is smaller than Australia, I believe, and Ravenloft's main continent is about the size of France. (The one part of the setting outside that that is really big is the Russia/Siberia analog, where vast frozen wilderness is sort of the point.)
Again, I suppose. Although, "smaller than Australia" is hardly small. It is a continent after all. And, while Ansalon might be that size, there are numerous other lands as well.
To me, Ravenloft got it right. A campaign setting should be about a month's travel side to side. For me anyway.
OTOH, if it's that small, you have a much more limited choice for terrain and climate.
I guess I just prefer much more focused than diverse in my settings.
Totally a personal preference there.