Gamma World would be "safe". It's really not much more than adding psionics and an post apocalyptical setting. Easy, low risk. It would be an evolutionary choice. This could enable something like the settings and worlds from Shanara or even Mad Max.
Star Frontiers would be revolutionary (relatively). I too have a strong emotional attachment to this one, and still play it bi-weekly. The setting is very strong and compelling. The original rules are weak, and the Zeb rules are arguably not an improvement. But, that could/would be solved with a good conversion to 5E. I will note their are a couple of fan 5E conversions as well as the RPG Frontier Space which is very siumilar in feel and can be used to run a game in the SF setting.
SF would also put a good damper on SFRPG from Paizo, though Star Frontiers would sort of share the same acronym and could be undesirable. But they games would be drastically different.
Well done, I would enjoy either. And, as for the printing press that is D&D, I don't see a new 5E ruleset impacting that. Unlike Greyhawk etc, there would really not be any splitting of the fan base, but rather widening and capturing a broader market that already exists and they don't compete in.