You can't backstab a column of stone. If you apply enough force to exceed it's ability to resist shear, it will snap in half. But this is done by applying massive amounts of force to it over time...not by chipping at it. If you're chipping away at the column, it doesn't matter where you strike- as long as you strike the same spot, you'll make it through at the same rate you would anywhere else on the column.
I think it's fairly clear that adventurers fighting a golem are doing more than just knocking chips off. If that were what they were doing, it'd crush them long before they dealt any noticeable damage.
And that is what most golems are- undifferentiated, solid matter- animated by magic. They are not articulated like action figures; they have no joints; they have no internal frame, just solid matter. They are animated statues. The same magic that animates them also keeps their jointless bodies from disintegrating due to structural fatigue.
Exactly what the magic is doing is not entirely clear, but regardless, a statue most certainly has weak points. There are crevices where force could be applied to split the stone; there are narrow places like neck and wrists where a well-aimed blow might cause a break.
IMO, the place where complaints of unrealism apply to golems is much more basic: You can kill them with nonmagical weapons*! Flesh golems, okay. Clay golems, maybe. Stone or iron? Please. Try whacking a stone statue with a sword and see how far you get. You'll break the sword long before you damage the statue.
[size=-2]*From 3E onward, at least. This is one area where I think AD&D's "magic weapon or go home" rules make some sense.[/size]