I haven't tried many different RPGs but I can think of a few things that would break the deal right away:
1. Over-design and over-build of characters. If I can't roll up a playable character in half an hour with little-to-no previous system knowledge: see ya.
1a. If I have to choose between flavour and mechanics - e.g. if I read the book and immediately see that if I want to play a character with personality or abilities x I'm going to get screwed by mechanics y, z and a: see ya. (3e multiclass spellcasters are a good example though it took me a while to learn this one the hard way)
2. Over-reliance on a single core mechanic. Core mechanics are fine provided some attention is paid to when they work and when something else would do better.
My core deal breakers:
1) Randomness in character generation - No element of my PC should ever be random. EVER. No random stats, no random HP, no random lifepath table, etc. I want complete control over every aspect of my PC.
It's a good thing we're not on the same design team as this argument would rage through the nights...
I want some random. Not everybody is the same in real life, nor is anyone perfect, and I want both the system and the game world to reflect that. And I can play whatever the dice give me...
4) Anything that prides itself on being "gritty". I want to play a big damn hero! I want to be capable and badass at level 1.
Where I don't at all mind working my way up; but this one is easily solved at the design level by having a series of possible start points - your level 1 might be my level 5, for example; and you just change the level numbers to suit. The game system just has to tell us what numbers to use for how badass we want to start out as.
... especially if they are your PC's main source of power/coolness.
This brings up another deal-break for me: power and coolness coming from mechanics rather than personality. Sure it's fun to bust out some neat combat trick once in a while, but the real joy is being the coolest guy in town by roleplaying that you're the coolest guy in town, and doing it well enough that people buy into it.
The most mechanically bland character in existence can still be the coolest guy in town, if he's played that way.
Lan-"not the coolest guy in town but possibly the loudest, and certainly the coarsest"-efan