I think a good game can have both. My preferred game has both.
Small dungeons are set areas to be cleared and dealt with. There are a lot of them in the world -- even a deserted house qualifies in this sense. They give a definite sense of completion when done, and they give a definite sense of parameters to the PCs...the PCs know that the upper levels of an old tower go so far, and no farther.
Large dungeons are more like wilderness. You are not usually trying to explore every room, which may be literally impossible. Often, you are trying to get from Point A to Point B. Sometimes you need to make contact with a denizen of the complex (like the lich who lives in a ruined city) for aid, information, advice, or support. Sometimes you are seeking a particular McGuffin or Foe, long lost to time. In sandbox play, the large dungeon also means that there's always something to do, even for pick-up games with little or no prep time.
Many small ruins and a few great ruins also make a game world seem more real...to me, at least. There may be thousands of small cave complexes for every Mammoth Caverns, but I'd rather play in a world where every small cave complex might turn out to be Mammoth Caverns when you go a hitherto-unexplored way. I'd rather DM that type of world, too.
RC