I feel like this is just such a bad read of things. You're talking about art style, not substance,
I'm talking about how you communicate your product to potential customers. The first thing most people interact with is your visual presentation. If your game is about a Madmaxian post-apocalyptic wasteland or spacefaring fighter jocks blasting frog-people back to the black hole they came from, presumably, your box art will not feature a bearded dwarf wielding an axe against dragon.
What's this say to somebody who doesn't know anything about Pathfinder? It says, "I'm a knock-off of Dungeons & Dragons." If I'm already happy with D&D, I'm uninterested. If I'm not interested in D&D, I'm definitely uninterested. If I'm unhappy with D&D, depending on why I'm unhappy, I might still be uninterested (I might be sick to death of gangs of medieval murderhobos killing ogres and dragons)
. The presentation screams that this is is a product designed to appeal to sophisticated RPG consumers who already know who Paizo is. This boils down to people who already love Paizo and will buy whatever they make if it's any good, and people who love D&D themes and want a new mechanical twist on them. It's extremely limited, because the market of "D&D fans who feel like the latest game from WotC completely lost the plot and fails at meeting basic expectations" ain't there no more.
and your point is just that much with GURPS because it has a mold-breaking setting: Infinite Worlds. That whole setting of multiversal intrigue and adventure where you can fight world-hopping mirror fascists, supertech Nazis, modern wizards, sentient dinosaurs, a Rome that never fell, superheros, and anything else you can think of. That's totally not regular dungeon fantasy and it's honestly awesome: I adore that book and everything in it.
This sounds like a commercially terrible idea that sold horribly. [Reading your next sentence, it was.]
Your mistake is saying that the market views "fantasy heartbreakers" as D&D clones when, really, every RPG is viewed as a D&D clone. To take a comparison from another thread, D&D is the Kleenex of the RPG industry and basically everything in the industry is defined by it. "D&D but space", "D&D but the 1980s", "D&D but cyberpunk", "D&D but _______" is how things are viewed regardless because for most people, D&D is RPGs.
I don't think VtM was viewed as a D&D clone in the 1990s. Granted, a lot of people saw it as "goobers reading poetry to each other between dice rolls," but it was definitely different than spelunking dungeons to get loot.
As to Starfinder, I dunno. It doesn't interest me, but I suppose it's gotta interest some people as it seems relatively popular. Beyond Star Wars, what other setting are out there that might be doing better? Traveler? Phase Eclipse?
Starfinder interested me right up until I found it was going to be, thematically speaking, Off-Brand Forgotten Realms in space. I'll allude to something else I said. Go back to when D&D got big, and you see it's riding a wave of sword & sorcery popularity that is as much fueled by everything from Conan the Barbarian to Masters of the Universe as anything TSR was doing. VtM was successful for the same reason; you had that kind of dark, gloomy ethos everywhere in the 1990s, whether we're talking about Trent Reznor tearing up the charts, or movies like The Crow. I think if you were going to try to build success off something truly different, you'd try to build off current pop culture trends.