If a friend makes a joke about, well, anything, I don't really care. Cuz it's a joke - I cannot stress this enough. Jokes are different from mean spirited attacks.
Saying something is a "joke" is not an excuse.
"What do you call 1000 dead [insert racial slur of choice]? A good start!" is a joke, but it's also offensive.
It's possible for the underlying context to change the meaning meaning of something; a racial slur can convey different meanings if it is used to intentionally portray a character as a villian, or if it is use to display a juxtaposition of different positions, or if it is used as a refuge in audacity. In the case you give, the performer and audience (you and your friends) is the context that makes a ginger joke acceptable within that group. But saying something is a "joke" is not sufficient context to make a blanket statement about how offensive it is.