The resultant golem is 100% animated by magic. However, since the physical form is the anchor for the magic, enough damage to the physical form will disrupt the magic...just like erasing part of the magic circle that holds an extraplanar being in place or at bay means the barrier falls.
The only difference is that a golem is far more durable than a magic circle.
This doesn't change that
it still has structural weak points
What does a spine do? It supports the rib cage; it is an anchor point for things like hips or shoulders; it protects the spinal chord.
...none of which a golem has, because there is nothing inside a golem except solid matter or trapped air (depending on its construction method).
Uhhhh.
The spine does a lot more then that.
Look, you keep missing the main point:
If it stands. If it
exists. It has a structural weak point. It has nothing to do with magic; if it is made up of a structure, it has a weak point in that structure, especially as an animated statue.
A construct is just that - it's
constructed. It has joints. At the end of the day, regardless of how they're animated, golems need legs to walk on and arms to punch with. They still rely on the musculature of the corpse / statue to move and act. Skeletons have joints. So do constructs. Hit anything with a form and structure at the right spot and you can do more damage than hitting it in a generic spot.
Look at it from a structural engineering standpoint.
Give any jerk enough C4 and a working knowledge of how to arm and operate a radio detonator, and he can bring down a building. This is analogous to the musclebound fighter or barbarian with a sword the size of a door and Power Attack. He swings that thing with so much force that he's going to get results. Sometimes, his swing connects in just the right spot and he really gets results (critical hits).
But a demolitions expert can bring that same building down with a far smaller quantity of explosives. Why? Because he knows where and how to place them to collapse the load-bearing structures of the building, rendering it incapable of continuing to stand. This is our rogue with a dagger and a heart full of hate. So long as he's in a position to really choose his target, every strike is placed into just the right spot to get the job done with minimal force and effort.
Although a stone golem may be a single contiguous mass of granite, certain points on its body bear its weight more heavily than others (I'm looking at you, hips). Bring enough stress to bear on those sections, and the thing can't support that weight anymore.
Of course, at the end of the day, this whole argument is about HP, easily the most vague and abstract thing that has ever existed in D&D.