When you can't meet your own deadlines, when it interferes with actually running the game, or when your players get irritable.
Let me relate my "line". In getting ready to run my current game, I thought I'd adjust all the things I disliked about D&D, or those that just didn't fit my setting. I loathe fire-and-forget magic, so I decided to remove that -- which led to needing to change any class that cast spells in any way (which excepts only Barbarian, Fighter, and Monk) -- including making the Cleric more religeous and less martial. I wanted to add a class defense bonus. I wanted to create an entirely new pantheon of gods -- and a cosmology to house them. I also wanted to have a rather large continent or two (Eurasiafrica). Plus I wasn't entirely happy with any of the core races besides humans (some only needed minor tweaks -- elves have a favored class of Ranger).
I putzed with it for six to eight months, but without hurry. Then, the RtToEE game we were playing ended up a TPK for the second time, and was ended. So, I had about a month and a half to get things going.
Well, with a 40+ hour a week job, a wife, and two kids; I discovered that I just don't have the time like I did in high school and college. I started figuring out what
had to change and what didn't. I also looked at what could be done after the start of the game.
With the spell point system in Unearthed Arcana, I added that and ditched sorcerer as a class. I dropped the Cleric and used
Hong's Priest class. I also converted both the Priest and Druid to spontaneous casters.
I copped the Defense bonus from Wheel of Time, and adopted a few other options from UA. Basically, I found that -- without delaying my game way too long, quitting my day job, or selling off my wife and kids -- I just didn't have the time to rewrite the rules. I've effectively left that part of my brain to handle the selection and management of published variants from other sources.
To sum up, the "line" come whenever the banes outweigh the boons. Those banes can be lack of interest, frustration, or simple lack of time (I certainly still enjoy tweaking the rules, I just have other priorities).
BTW, if anything I've done can help, feel free to swipe it. <pimp>
http://www.users.qwest.net/~burkettjd/Gaming/index.html</pimp> Right now, it's just the compilation of classes modified for my house rules, but I'm in the process of posting more stuff.