An setting may be believable in a "scientific" sense. That does not mean it follows real world's science, but that it consistently follows its own laws, laws that can be discovered by someone who lives there.
<snip>
A setting may also be ruled by laws of its genre; laws that treat it as a work of art. Indiana Jones, Star Wars or Pirates of Caribbean have very little "scientific" consistency, but are fun anyway. We don't ask how is something possible, or why a character acts as he does. We ask what is appropriate, interesting and fun in this kind of story.
Perhaps I've misunderstood or perhaps I don't know how to apply it, but I don't quite agree with the way this was worded.
I'll use the example of the recent Transformers movies. The law that alien machines can transform is an internal "scientific" law. The law that two fighting robots will utter one-liners like "Now you die, Optimus Prime" is a genre law. Neither of these bother me; I'm OK with that.
But if you were to imply that most other bits of silliness in the Transformers movies are also excused by genre laws; this is where I'd disagree. Bad plots and inconsistent character behavior is not a genre thing per se, but just lazy writing.
Then there's The Matrix. I don't question the "scientific" law that those who've taken Red Pill can escape the Matrix through a land-line telephone booth. I can accept the genre law that people can frequently outrace machine gun fire. Everything is just so well-written and well-thought-out that I am totally immersed in the experience.
I think that a "bad" unbelievable RPG game is like Transformers: entertaining but dumb and silly.
A "good" believable RPG game is like The Matrix: entertaining, well-written, clever and consistent in its internal logic.
Trying to apply scientific analysis to this kind of setting is an exercise in futility. Applying literary analysis, on other hand, works. In this kind of logic, an answer to "Why can't I trip an opponent more than once an encounter?" is "Because it would be boring if you did it all the time."
In that specific example, I think the real answer is not that it's boring to trip opponents all the time, any more than it's boring to use any at-will power over and over again. I think the answer is that the developers don't care if anyone asks the question "Why can't I trip an opponent more than once an encounter". If they did care about that, they'd either change the tripping rules (not, not to make it more complex, just different) OR offer some fluff to justify the rule (DM: "Your opponent doesn't fall for the same trick twice in a row and learns to evade your 2nd trip attempt").
As for the large picture of un/believability stemming from game rules, I'd use this analogy. In Transformers, they bring the All-Spark into the middle of a populated city and the climactic battle endangers the lives of its citizens and causes billions of dollars in damage to infrastructure. The decision to bring the All Spark to Mission City was utterly reckless and ridiculous. The internal logic is neither defensible nor relevant, really. The REASON they brought the All Spark into the city is because the writers wanted an explosive-y battle in the middle of a city, and they couldn't think of a better plot device. The question of internal logic is irrelevant because the ultimate decision was external to the fiction.
Equivalently, many metagame rules ruin the believability of the in-game experience when the metagame priorities are so high as to override the internal logic of the setting.