So far, I've heard lots of speculating & abstract theorizing & still no indication that anyone has found out what was said, exactly. I have some inside info from someone who was watching the panel. Because I know this, I believe two things are likely to change if more people knew it, too: 1) Opinions will still be split; 2) The split will likely shift. In short, it does matter. You should know more of the context. The author's apology & explanation did not contain these specifics. Neither did the spouse's objections on Twitter. Here's what I heard. It's hearsay, when you hear it from me, but it's a quote from an eyewitness.
"She was describing what a great author he was, explaining how influential his work was, mentioned all the awards he got, etc. etc. and then said it was all the more remarkable because he had to overcome hardships because he was a colored author."
Now, I will make my comment, based on this. Admittedly, I'm interpreting. In this context the speaker is not quite saying precisely that this fellow author ought to be characterized as colored, but that the fact is that there were people who knew him & could have an opinion of him, and some of them would've considered him to be colored. In the past, that would've created various forms of bias, conscious or unconscious. That likely would've caused problems for him. Which we, seeing it, would consider to have been hardships to overcome. In light of which, we should see his accomplishments as even more laudable & worthy of our appreciation.
A perhaps more careful phrasing, with my interpretation, could've been "he had to overcome hardships because he would have, in the early part of his career, been thought of as a colored author."