I'm wary of any rule that:
seems complicated
seems to drastically change the game
tries to make it "more realistic"
makes your PC more powerful
That's not a bad checklist.
With respect to the first, a finely honed Occam's razor should be in the toolkit of every prospective game designer, including aftermarket designers (aka referees) - think simplicity and elegance.
With respect to the second, I tend to think of myself as a rules 'tweaker' - a number of my house rules consist of fixes to corner cases or add-ons to existing systems rather than whole-sale creation of new subsystems. I feel that the less a gamer familiar with the system needs to unlearn-and-replace, the easier a house rule is to integrate.
With respect to the third, this depends on a couple of things. Every game abstracts the physics of the game-world through its rules, and I can understand why someone might be tempted to fiddle with the granularity dial if the rules-as-written produce results which fail the smell test for whatever they're intended to simulate. The key here is to understand how twisting that dial affects other systems: character creation and improvement, combat and skill performance, rewards. It's possible to make a change to one set of systems but not consider the cascade of effects of the change on others - this is why playtesting and critical feedback are important.
But tweaking a system to enhance the unique qualities of a setting can, if it's done well, add considerably to the experience of the game, so I don't think every attempt to tweak the abstraction dial is inherently game-breaking. As
irdeggman notes upthread, many or most house rules are intended to reflect some aspect of the game-world; many of the
house rules for my game take abstractions in the rules intended to emulate 17th century France and make them more specific to the 1620s where this game is set. In most cases my house rules open up additional options for players; only one specifically narrows a player's option with respect to character creation.
With respect to the last, I think it's important to understand and respect balance without being shackled to it, and to consider any change in light of how powerful the adventurers will be
vis-a-vis both their opponents and one another. This is another instance where playtest, playtest, playtest should be the designer's mantra - frex, take a half-hour each game-night to set up a side-game specifically for testing your house rules before you introduce them into your long-running campaign. Get feedback, from the players in the playtest, from on-line forums, from the designer of the game.
Both professional and amateur for-market designers are capable for turning out complete stinkers as well, even with what should be a more intimate knowledge of the system; I reject the notion that aftermarket designers are inherently inferior in their command of the rules.