I've been thinking about this a lot and have started building something around it. Curious whether others have developed any reliable signals or methods for compatibility before session one, or whether it's mostly still vibes and luck.
I also use my home to run and play in games. For me, it is about screening prior to that. I played in a short campaign. It was one where we were all paying $25 a session to play. Some people at the table I kept, others I didn't. I think this is key. The group I had, knew that before inviting anyone over, they needed to ask. Most of the time, no problem. If they were to join our group, it warranted a discussion among all of us. This took almost a year to get to a great place.
But even now, there will be complications. New players want in because they are a friend of someone at the table and think it sounds fun, and we have to decide... Is this going to work? Then they show up as a "trial" run for a few sessions.
I have said no several times to co-workers. It's painful.
We are now capped at seven players. And for some of us, that is too much. But, we all get along, have fun, and enjoy the same playstyle. And in part, that is because of the screening I did, and later, we did, early on.
There have been other times I let people invite me, and I decide if I am a good fit for that table. I am very fortunate in the fact that almost everyone I have played with through a campaign has been good. I am lucky, and I understand that may not be the norm. But for me, the coin has just always seemed to land on the right side.