Right. I guess my mistake was thinking that, in your scenario, the golem's armor would be taken care of in the AC. That is, AC is not only hitting, but hitting parts that aren't armored/hard enough to make a dent/etc.
Perhaps more concisely, if your attack roll meets or beats their AC (or...