No, I'm pretty sure I'm understanding you.
The fact that you even feel the need or desire to say this is pretty telling.
"You can indulge in <something that doesn't affect me> when I'm not around" is not a reasonable statement. It's the default state of adult relationships. Similarly, "You can indulge in <something> that <doesn't affect me>" is also the default state of adult relationships.
By contrast, "You can't indulge in <something> that <doesn't affect me> because someone who indulges in the future or has indulged in the past might or has affected me" is controlling behavior, and in the context of a leisure activity like a game between friendly acquaintances it's peculiar.
Sometimes there are good reasons to impose controlling behavior (prior restraint). All of these that I can think of involved concerns for public safety or the unknown factors of dealing with strangers. As far as i can tell, unless we're misunderstanding your situation, your controlling behavior is simply a matter of your enforcing your preference on someone else because "maybe."
Very lucid -- this summed up my (more visceral and less intellectual) take on things after reading many more oblique posts and responses.