I would argue that even that is a real big stretch for a lot of people because a critical plot point of Starfinder is that Golarion has been completely removed from it. Indeed, that is supposed to be the eternal unsolved mystery of Starfinder: where the f#$k did Golarion go, why does no one--apparently not even the gods themselves!--remember anything about its disappearance, and why is it they do know that Golarion hasn't been destroyed or harmed despite being totally inaccessible?
It's hard to call Starfinder "sharing the same setting" when literally the entirety of what makes it the world of Pathfinder has been neatly excised. I mean, I guess there's the rest of the star system Golarion used to be in, but that's pretty slim pickings. You could just as easily assert that Star Wars and Star Trek "share the same setting", they just occur in different galaxies of that single setting, making any actual travel between them impossible even though you could--theoretically--observe one galaxy from the other.