I never said anything about total modifiers, only the penalty to a creature or object's Defense based on size. Let's face it... an unattended object or structure has a Defense of 5 + size. So, a Colossal or larger building has a base Defense of 5 - 8 = -3. In order to miss with a ranged attack against this object, you'd have to either be carrying some serious penalties (possibly for range) or roll a natural 1. Imagine yourself adjacent to it and using a ranged weapon. That gives a +1 bonus to attack rolls (all ranged attacks against adjacent squares gain the benefit of the Point Blank Shot feat), so in order to hit with anything but a natural 1 you'd have to already have a -5 penalty to your attack roll. If you're carrying penalties like that and roll so low, you probably shouldn't be hitting anything of any size. Like I said, "If you can't hit the broad side of a blue whale, you can't hit the broad side of anything."
Of course, there are some exceptional circumstances that can negate this. For example, if you are on the inside of a gelatinous cube, a monster's stomach, a mountain, or a building, it's pretty hard to miss if you're not targeting a specific part of the interior.