Man, you guys are lucky, or else you have easy standards for friendship.
I've got a couple guys in my gaming group that I do things outside of gaming with on a regular basis. I've got a couple more guys that I like but pretty much just game with -- we just have pretty different lifestyles. They don't wanna come over and watch football, and I don't want to go eat sushi. And there's one guy in the group that I don't dislike so much as I don't understand him -- his motivations, what he likes. He rubs me the wrong way, but he's also the friend of three of the other people. Because he lives where the game is, there's no real way to get hiim out of the group, and because he's friends with the others, there's no real way to get him out of the group...

Even though the others have said, "yes, he's like that, that's kind of an issue he has," it's just something I put up with. As I'm sure he puts up with me.
If it's somebody I loathed, that's another thing. Unhealthy is unhealthy. I just left a writing group because of a situation that was actually unhealthy. But only gaming with my lifelong bestest friends would have me, well, not gaming much. Or gaming and asking people to choose between me and their other friends, which is not always a cool thing to do.