Game theory (as a term) is specific to a branch of math, not to critical theory about games. There is a burgeoning field of videogame critical theory, however.
While game theory is certain a math derivative, the concepts could apply. Particular quantum game theory. Mind you that's a whole thing that is just getting started.
Remember, that game theory is also about decision making and logic, both things that can apply to an RPG. That said, I don't think its the best existing theory to look at for inspiration.
No. I would dispute that. Settlers of Catan is a board game; a slightly complicated one, one that involves strategy and is not "perfect information" nor "deterministic" (very specific terms), but is still very different than the heuristics and critical theory one would apply to an RPG.
Finally, while there are basic approaches to videogames (say, which are much closer matches than boardgames), even these are necessarily restricted within specific areas; comparing the examples of Journey, Candy Crush, Binding of Isaac, Call of Duty: MW, Civ V, Ghost of Tsushima, the Return of Obra Din, God of War, and Tetris Effect should indicate pretty quickly that the qualities that are necessary in one game are not necessary (or sufficient) in another.
What works for a LARP may not work for D&D may not work for BiTD may not work for another RPG.
Yes Catan is a board game, but it is still a game, played as a group at a table. At its very core is the idea of a group activity governed by rules easy enough to be expressed in a shortish book. Like and RPG, now that isn't to say they are the same, because they aren't, but my point was that two games have more in common than a game and a novel, even if one of those games emulates the genre of novel. I recognize that Catan isn't chess any more than D&D is chess.
I do agree that for the reasons why RPGs are played at all video games have some critical thought on that topic already, and since they're interactive the categories tend to work well.
The Angry GM has a pretty good break down of the categories and how they apply to RPGs vis-a-vis video games.
To clear though, you mean deterministic in the way one could in theory play the same game of chess over and over as the are only so may prescribed moves, and using the same ones every time will end in the same result. And perfect information in that the players of chess can see every move and know what their opponent can do and has done at every turn. Compared to Catan where there are hidden cards, and randomness via dice. At their core RPGs are stochastic games with imperfect information. In a lot of ways RPGs are Assymetrical-Super-Poker-on-Steroids. The game is one of stochastic perfect information for a GM (where one exists) and stochastic imperfect information for the players.
In the end my point is ultimately that in the G part of RPG part is the most important part (either the game or grenade, take your pick). In fact, I learned something about terms for categories of games without talking about RPGs directly, since they're still games and fall into broader categories which have specific terms that can apply.