Most of my house rules, even the involved systems, started off as one-time rulings where the RAW didn't seem to reflect the game fiction well. Additions and refinements were added as needed for in-game situations. Growing house rules organically this way helps to avoid problems since each small change is tested along the way and can easily be reverted if needed.
One of my famous flubs was a House Rule I implemented in my 7-year WEG Star Wars D6 campaign that I ran back in the 90's. If you asked my players to vote, I think they'd all call that one their favorite campaign we've ever played of any rpg.
I saw an optional rule in a sourcebook or SW rpg magazine--I've forgotten the source--where a character can take dice off of his attack and apply it to damage, making the attack harder to make, but damage quite a bit. This is not unlike the Power Attack Feat in 3.5 D&D (which I've seen have it's own controversial discussions of power creep as a character levels).
We had a math major in the group who would bring probabilty tables to the game to share with the other players, since figuring the probability of a task when using multiple D6 only can be complex. Before long, the players had a general rule. They knew when to move dice and take advanage of the rule, and when not too.
Now, all of this is fine with me, except that I noticed that the game's design was based on this sort of thing not happening--this was something I realized only after playing the game for a long while, getting to know all of the mechanics' subtle particulars.
I knew that whacky things were happening in the game. For example, the characters were extremely powerful in hand-to-hand combat because, at close range, he difficulty to hit was lower than when a blaster was fired. This allowed more dice applied to melee damage, with the effect of fists and vibroswords doing a lot more damage, on average, than blaster pistols and rifles. It was especially bad with the extremely strong. We actually had a Wookiee run up to an AT-ST and rip one of its legs off.
Another unintended consequence was that, since it was a heroic game, the game, itself, had built in rules to make the PCs better than the NPCs. That way, when you've got your Luke-type PC fighting a dozen stormtroopers, Luke wins. Well, with the dice swap rule, Luke just didn't win--he annihilated.
Of course, the players LOVED IT! And, I remembered the excellent advice that E. Gary Gygax gives in the 1E AD&D DMG, saying that, basically, players will always be pushing for more power, more ability, better equipment, and it's the DM's job to rein them in, in order to keep the game fun and challenging. A player would love to start the D&D game with his mage having a Staff of the Magi, but it's the DM's job to allow this later in the game, after the character has earned it, avoiding power creep.
I tried to revoke the dice-swap rule, but, to a man, my players (seven, at the time) all mutinied on me. We had one game session without it, and I was forced to live with my mistake because the players liked the rule so much.
All of this happened because, as I read the rules, I saw the dice-swap option and said, "Hey, that sounds cool!" I introduced it well. It caught on. And, I couldn't change it, from then on.
I learned that a DM can't give a player a Staff of the Magi when his character is created only to take it away from him later (maybe a bad example, but I've removed lots of over-powered items in games through story-means, with the character losing it, or it being damaged somehow, or running out of power/charges, etc.). The dice swap thing was a game mechanic, and I had no story-related gimmicks to fall back upon.
To this day, I've learned to play new games by RAW for a long, long while. Get to know the game. And, then, slowly...ever so slowly....change things after much thought.
That's why you'll see me talking about rules on this forum. I usually ponder things for many months before a change is made, and I try to keep all changes to RAW at a minimum.
I work with one of my players, and we actually were talking about old times last week, and the Star Wars game. He said, "Man, that was a great game until you tried to screw it up!"
He meant me trying to take away the PC's uber power with the dice swap. He still can't see how it hurt the game in the long run. All he knows is that, when he played the Wookiee, he ripped the leg off the AT-ST, and he enjoyed the heck out of that.
If I ever play SW D6 again, you can bet that I won't have that rule (well...maybe as a Force power, or something...not a standard rule for everyone). And, my blaster rifles will be feared again, not the dagger in the Wookiee's hand (well, anything a threatening Wookiee does should be intimidating, but you get the idea).