When it comes to homebrewing with 5e or writing 5e 3pp, at what point in changing the rules, genre, or mechanics of 5e is better just to create or use another system entirely than 5e? At what point does it become another system entirely and should be distinguished as such?
I can't answer for 5e specifically, but I play a homebrewed version of 3.0e that is at least as different from 3e D&D as Pathfinder is from 3.5e D&D.
I will say that writing up a completely novel rules system is much harder than just amending one to taste. The process of amendment is an incremental one, which at each step of the way leaves you with a playable game. When you go about creating your own entirely new system, if your ambitions are large then you have a year or more of work ahead of you before you even have enough to present to would be players. And you'll still need to go through the process of playtesting the rules and amending them on the basis of lessons learned.
Likewise, I'm of the opinion that the world really doesn't need a brand new RPG system at this point, and given the great depth of time and effort spent on making rules systems, there probably is no potential rules system out there worth reinventing the wheel over. The history of RPGs are littered with hundreds of fantasy heartbreakers that are often no more than tweaked versions of D&D anyway, and the majority of novel game systems are often inferior to old standbys anyway. After several attempts to create rule systems myself, that resulted in unworkable messes once I added all my cool ideas to them, I've largely given up on that myself. There are maybe a dozen solid RPG systems out there - D20, BRP/Pendragon, D6, DitV, Cortex Plus, WOIN, etc. - that can be tweaked to do pretty much anything you could want to do in any setting you'd want to do it in.
So probably the answer is, "Is it actually impossible to tell the sort of stories you want to tell in 5e without tweaking it, and is the amount of effort that would take greater than the amount of tweaks it would take to get a different system to where you'd want it?"
At what point does it become another system entirely and should be distinguished as such?
When it uses both a radically different CharGen and a radically different fortune mechanic or radically different process resolution, then it's probably a different system. If it tweaks only one process in that, it's a different game but it would still be in the D20 system of games - say Mutants and Masterminds or any other True20 game. (Although to be fair, True20 tweaks enough that I could accept the claim it is a different system.) If you make tweaks but you still have classes, levels, hit points, and a fortune in the middle resolution process that depends mostly on the outcome of a D20 plus modifiers, then it's still basically D&D reskinned for a slightly different setting or style of game.
Of course, I don't think that really matters too much, as there is no way to measure really how far you've moved from house-ruled X, to brand new game Y, to brand new system Z and largely that doesn't matter except for conveying how familiar a system is going to seem to someone who knows X. IMO though, Pathfinder, D20 Modern, Star Wars D20, etc. is basically house ruled D&D. M&M or 13th Age is a brand new game based on the D20 system, and PbtA is a brand new system.