I don't think that I would label licensed IP as a whole as a "failure". Games come out, they live for a while, and they die - licensed or not.
The biggest challenge with a licensed game is just that - there is a license involved and so there is an extra layer of expense added on to all the usual challenges of publishing a tabletop RPG. Those fees tend to go up, not down, and at some point even a successful line will hit a breaking point.
Not sure how FFG does it but they've had a pretty comprehensive Star Wars RPG in print for about 11 years now (12-23), same edition, consistent mechanics. WEG did it before with a roughly 12 year run (87-99) that covered everything that had been released during their time. WOTC had a similar run of about 11 years (00-11) but released 3 different versions of the game as each of the prequel movies came out. I don't think anyone publishes a game line for over a decade and calls it a failure.
I've run and played multiple short and long campaigns with Star Trek, Star Wars, and Marvel RPGs and they are a ton of fun but they are all very different from a typical D&D campaign. If you can find a group of players interested in playing the crew of a federation starship your Trek game will probably go well because that's the baseline expectation in those games. Not everyone wants to play that though. Or Star Wars. Or Marvel superheroes. Even when one or all of these have had a really strong system you have to want to invest time and money into playing them.
In fact none of these properties dominates much of anything other than being somewhat popular movies or TV shows:
- The biggest miniatures game out there is Warhammer 40K, an original IP. There have been many Star Wars games, some Trek games, and even some Marvel games, and none of them have managed to take the #1 spot.
- The biggest western MMORPG is World of Warcraft (another original IP) and has been almost since it's launch in 2004. There have been 2 Star Wars MMOs - one of them failed years ago and one never met expectations though it is still running. Star Trek has one MMO which is alive but very small. Marvel had one MMO game and it was never huge and was shut down years ago.
- Star Wars occasionally makes noise with a videogame release but the biggest ones are still things like Fortnite, Call of Duty, Dark Souls/Elden Ring, Madden, Fifa, and various racing games. Marvel has one out now that seems to be pretty popular but it's one game. Trek hasn't had a big game in some time.
I think expecting a big franchise to take over RPGs is a reasonable thing at first sight but if it isn't happening in other genres maybe it's not. Reasonably popular but not dominant seems way more likely and I'd say closer to how it has been historically.