I've never quit a game because of one person, but then I usually DM. I have, on the other hand, scuttled an entire gaming group because on the first game session, a guy I had invited brought his girlfriend and other friend to the game on short notice and I couldn't stand either of them.
We were starting up a Darwin's World game and among the annoying things: the girlfriend frequently stopped play to ask for irrelevant details that were not provided (it was a published adventure), then criticized me for not providing enough detail, had her character stop off to load up on feminine hygiene products, generally annoyed the crap out of me. The best friend was a hyper lunatic who wouldn't shut up about his other character in another game who was just soooo uber kewl. At one point both of them criticized the fact that we were playing a game other than D&D. Hello, you two weren't even invited in the first place, so I could give a crap that we weren't playing your game of choice. Arrrgh! I ended the game early for the night, right as I was filled with the urge to kill not only the two losers, but the guy who I had invited and felt the need to drag those two along.