Well, I think it's easy to get caught up in assigning a detailed alignment chart for a fictional character, alignment is just one factor determining why somebody does what they do.
That's true. Characters also have personalities, preferences and other sorts of dispositions.
But respecting that other people should also be able to choose for themselves is part and parcel of being chaotic.
True, but only fully true of True Chaotic. Chaotic Good draws a hard line on choices that cause woe, and will even intervene if the choice is only causing woe to the individual making it. They are balancing Freedom with Weal, and electing to have an active role with respect to questions of Woe and Weal. They may want to live in a society with unlimited and untrammeled freedom, but they may realistically only see that as possible when everyone has transcended evil impulses. Chaotic Evil on the other may believe in unlimited freedom for themselves, but think that not only does that freedom not extend to others, but that it should not extend to others. They believe in a zero sum world were the only way to profit is at the expense of others, and where being heroic means being true to yourself and exploiting those that are weaker than yourself.
On good and evil, we are pretty close. Empathy I believe is neutral - a largely intellectual skill that doesn't necessarily define a goal. Compare the words Compassion and Mudita.
But you do get close to what I think is the core idea when you say, "not wanting to harm". My quibble there is that "not wanting to harm" is a passive state fully compatible with neutrality. A CN may "not want to harm". Consider the difference between the chaotic axioms: "Harm no one; do as you will." and "Do unto others as you would have them do unto you." The first is passive with respect to weal, while the second actively encourages it. Since the hallmark of neutrality is passivity or indifference to the axiomatic questions, the first I see as a CN manta, while the second is CG. So, what I would say is the core idea of good is closer to "Do the opposite of harm." Promote weal, health, happiness, prosperity, growth, etc. The ideal state of good is for everything to be infinitely abundant, and eternally healthy - growth and happiness and weal without limit.
Evil is the opposite, and again you touch on its central trait when you say, "objects with no inherent value". But I would take it one step further, evil - in its unmingled and pure state - doesn't believe
anything has inherent value. Lawful evil and chaotic evil find value in something - "the greater good" of the community or the wants of the self - but evil in and of itself finds no good in objects, others, or the self. All are equally worthless, and the process of evil is to prove this by removing any worth that things could be perceived to have - not just the degradation of others, but critically the degradation of self as well.