I don't want to derail the discussion, but...How does the term "War of Northern Aggression" relate to race or racism? I mean, it was white "aggressors" on white "defenders".
The racist part is that the war was over slavery. The South was for slavery (or for the right to practice it, in their eyes). Thus, the implication is that anyone who defends the South's stance (by flying a rebel flag, by saying the North was acting in aggression), are implicitly condoning or siding with the act of slavery. Because they are framing the North as the bad guys for attacking the south over slavery.
To Southerners, the Civil War and its symbols are a matter of regional identity and pride. Of stubbornness and standing your ground, and defending your territory, so to speak.