I think at some point you have to recognize that as much of a cult following as it has, spelljammer in many ways failed as a setting and that's part of why Planescape came along and became a dominant force in D&D a few years later and why spelljammer stopped getting developed and struggles to find a place nowadays.
While all I can offer is speculation, it seems to me that there was too much of a focus on using it as an expansion of lore and mechanics to integrate a bunch of generic fantasy worlds into the same universe and not enough focus on building it as its own campaign setting, and you'll note that's exactly what Planescape achieved.
That's what something like this would be an attempt to address to give it a more lasting appeal. My hunch is that WotC knows how much potential spelljammer has, but have been a bit stuck between a rock and hard place with it in part because of how they tried to incorporated it into 4th (which was terrible). They seem to have something up their sleeves, though.