First, I let it fester over the course of several games, usually punctuated by a lot of glaring. Then, at some point, I do something ridiculously passive-aggressive. That'll learn 'em. :)
Seriously, not much. I find that annoying people are a part of life- likely they exist in fantasy...