I agree that that answer is a fallacy. I'll just skip that response and go straight to "I don't care about this issue. I understand that you care, but your opinion on this does not affect me, and the fact that you care this much confuses me slightly", which is what I expect most of those...