The Curse of Ham is racist because it was developed into an ideology to justify racist oppression and is clearly and patently not the actual cause of differences in skin pigmentation.
Do you not think you’re being a bit hyperbolic here?
Historically, no, not hyperbolic. The Curse of Ham was frequently used in the United States as theological and philosophical grounds to justify slavery, preached from the pulpit and the podium. Whatever it was before that, in the American context, it was used too widely to support racism and bigotry to not have that connotation.
I think we should be more cautious when ascribing racism to something simply for sharing similarities else to something when it doesn’t share the actual racist elements that causes that thing to be racist.
This, unfortunately, comes down to, "the person speaking gets to say whether what they say is offensive," which is wishful thinking, at best.
This is a place where we should probably introduce the modern concept of the difference between a thing being racist, and it being bigoted. We can largely simplify it to: Racism is mostly about effect or impact. Bigotry is about motivations.
In D&D, the old story of the Drow is somewhat racist, even if the authors of that lore didn't intend to make it an analogy to African Americans, because it has outsized impact on folks of color, since basically the same story was used to justify the slavery whose impacts they still bear centuries later.
I could totally understand if the lore authors didn't get that - they probably lacked the perspective and life experience to understand, and didn't have guidance from people of color on the matter. But a great deal of racism in culture persists because folks don't understand, and don't consider how things may look or feel from another's perspective.
Calling this racist then isn't, "OMG, guys these writers were horrible people and soooo racist!!!1!" It is an error made in ignorance. Ignorance isn't a character flaw, and is correctible, so long as arrogance doesn't get in the way.