I've bowed out of three groups where there was a great GM, but the gaming was so bad, I actually dreaded going. In two of those cases, the gaming was bad mostly because of the system- it left an unsatisfying play experience. In one case, the other players were psycho, and better not to game than to deal with them.
Group #1- Two years ago, D&D 3.5 game. I never have really liked the 3.0/3.5 rules- it always feels unsatisfying in play to me. I've always found in the 9 groups I played 3.x with that the rules get in the way of play as people focus way too much on builds, min/maxing, system mastery, and "lone-wolfiness". The guy DMing the game was an excellent DM who made complex plots, deep characters, and had a lot of cool twists in his games (I'd played WHFRP, AD&D 2e, Alternity, and Paranoia with him, and always had a blast). The guys playing in the game all got along and were funny and nice. So I had high hopes for this game. Alas, it was not to be. This group devolved into munchtwinkery and the magical/character ability arms race to a worse degree than any group I'd seen(and I'm ashamed to admit, I did this too- mostly to keep my character even remotely relevant), and the DM was obviously frustrated they had done so. I bowed out of the game, and a few months later the DM ended the game because he was frustrated with the rules and how the players focused on rules to the exclusion of everything else. Oddly enough, the 4e group I'm running now had two of the same players in it, and they don't go all munchtwinkery in it. These guys also play in a Deadlands Savage Worlds game I run, with no problems. Rules can make a huge difference in player behavior.
Group #2- Probably 10 years ago, Star Wars d6 game- a system I really liked at first, but with more time playing it, the more I loathed it. Again, a good GM in that game, and the players were fun guys. It was set in the late Rebellion era, and we had 5 players: 4 "mundane" characters, and one jedi. What it came down to was that the jedi made all other characters irrelevant, except in very specific circumstances (computer repair for example). Playing backup for a character roughly five times more potent than any other character and who could do anything is about the most dull thing imaginable. I bowed out of that game, and although I've tried the Star Wars d6 system two more times after that, its always been with the same result.
Group #3- Probably 12 years ago, oWoD Vampire game. A pretty good system, but with its own quirks. The GM was a "colorful" guy, but very good- one of the only GMs I have gamed with who could weird people out during a game with his themes, mood, and subject matter. A truly "mature" gamer, and not in the retarded Book of Vile Darkness way. The problem here was the other players were truly the sterotypical mid-90's Vampire players (and yes, that means freakshow). I played with them for about five months, but eventually had to quit when a guy and girl in the group started acting really weird (and not grandma-friendly) during game sessions, and one guy who was schizotypal quit taking his meds and began bringing weapons to gaming sessions to act out what his character was doing, which ended with the evisceration of a couch with a katana.
