My initial guess would be Spelljammer, as those are probably the hardest adventures to adapt. They only did a few in Dungeon's entire run, possibly because they got flak for it.
If it was a more regular occurrence, Ravenloft maybe? They featured a significant number of adventures, and it's different enough from FR/Greyhawk that it would potentially impact sales. Plus Ravenloft adventures tended to get the cover art - Bane of the Shadowborn, Price of Revenge, Horror's Harvest, Last Dance, etc. Seems odd they would keep doing them if it was unpopular, but TSR was not exactly business savvy.
Man, now I miss Dungeon even more.
I'd guess Spelljammer or even Planescape. Planescape definitely has a cult following, but both of those are pretty nonstandard settings compared to traditional D&D.