The real question is have you talked to this person about this behavior and given them a chance to change, or even an ultimatum...
I am a non-confrontational person, so this is what I feel is right, but I think it is at best half true.
The fact is that the OP made a very good point. He has one afternoon while the wife is looking after the kids to play with friends. Why should he have to spend it with somebody he does not particularly gel with? Why should he have to spend it trying to deal with interpersonal drama? Trying to change a persons behaviours and attitudes is a lot of work. That is what some of us spend our 9-5 doing, I don't want to have to deal with that crap during my limited game time.
What does he owe the guy? Is gaming with some random guy for a few sessions enough to set up some fiduciary relationship in which the social contract demands that we spend time with them even if we find their behaviours annoying?
In the end, I personally would take the way of talking to him and trying to resolve it, but frankly I think that is due to a personality flaw on my part. A clear ability to confront people and cut them out of your life, without constructing social mores around it is enviable and would have saved me a lot of issues and time in life.