DMG, p. 65. Medium objects have 20 hp. Stone doubles that, and metal triples it.
Now, the others brought up excellent points where it would make sense and be consistent if you gave them some sort of "resist all" to reduce the damage, because those hit point totals are insanely low and mean even...