Being called out for being mixed-race is offensive to you . . . but is it offensive within Japanese culture? I honestly don't know. And is it being called out as mixed-race that is the problem? Or the use of the word "half" or "halfbreed"? I don't speak Japanese, but does the Japanese word you mention (ha fuh) have the same connotation in Japanese that "halfbreed" does in English?
If being of mixed-race in Japan isn't seen as a big deal, if the Japanese word used doesn't have a negative connotation within that culture . . . then those folks aren't being racist, not even in a systemic manner. Of course, if folks of mixed-race in Japan are looked down upon, and the word does have a negative connotation, then they most certainly are.
It's understandable however, if being referred to as "ha fuh" or "half" bothered you. It's reasonable to educate your Japanese friends and co-workers on the issue when dealing with those of mixed-race from the US.
I only point all of this out (from my perspective), to make the point that . . . what's racist and what isn't is complicated and the answer isn't always the same depending on the context.
Sorry, had some time away, so swimming a bit upthread to something that I think others addressed, but, this was specifically addressed to me.
I'd say that yes, absolutely there's no malice whatsoever in the people who were calling my childrend half. It's a pretty common term here for anyone that is of mixed Japanese/whatever heritage. But, that's kind of the point. It's a term that is specifically applied to people who are different from the perceived "norm". And, frankly, I would not really lean to heavily on expectations of cultural sensitivity from a culture that is so highly homogeneous as Japan perceives itself to be.
I think the point I was trying to make earlier about being "right" is never the important part of the conversation. It really doesn't matter all that much if something is specifically racist, bigoted, annoying, or whatever. Correctly labeling the taxonomy of behaviour is the least important part of the conversation. The important part is that this is offensive to someone (me) and it causes someone (me) distress. And, again, it's something that costs the other person nothing. It does not cost them anything to not call out my children as half breeds to their face. All it costs them is a tiny bit of restraint on their part to just... not say that.
Where the people come from isn't really the issue. My children aren't from Canada. They were born here. Go to school here. Will likely live here their entire lives, just like any other immigrant family anywhere in the world. It's not about dealing with mixed Japanese (and note, half only refers to mixed Japanese heritage) heritage people who come from another country. It's about not being called out as different every time you meet someone. Additionally, since it only applies to someone of mixed Japanese heritage - someone who was mixed Chinese and English heritage for example wouldn't be referred to this way.
Even if it's not malicious, and it's certainly not frothing at the mouth, epithet screaming racism, it's still a really bad look and it honestly and truly pisses me off. Heck, I don't even think it bothers my children as much as it bothers me if I'm being 100% honest. But, it's still something that costs people nothing to fix. So why not fix it?