Clearly, this is upsetting you in some degree, or at least is not fun to you. But this is a problem you can only solve talking with the players, not with the characters, or taking actions in-game. So when it happens, stop the game and present your concerns.
This kind of issues can be avoided...