According to the rules:
Elemental Spirit in Material Form. The construction of a golem begins with the building of its body, requiring great command of the craft of sculpting, stonecutting, ironworking, or surgery. Sometimes a golem’s creator is the master of the art, but often the individual who desires a golem must enlist master artisans to do the work.
After constructing the body from clay, flesh, iron, or stone, the golem’s creator infuses it with a spirit from the Elemental Plane of Earth. This tiny spark of life has no memory, personality, or history. It is simply the impetus to move and obey. This process binds the spirit to the artificial body and subjects it to the will of the golem’s creator.
Feel free to rule differently, but there are other creatures that don't make sense from our understanding of physics. Giants would collapse under their own weight, gargoyles would just be statues, a clay golem would just be a lump of clay. It's the elemental spirit that powers the form, no external magic is required.