For me it's the problem players and my patience for them. I've had too many hyper-munchkins, metagamers from hell, rules lawyers, and misogynistic grognards that don't listen when they're told to stop being so douche-y.
OK, so maybe my real problem is vetting players.