A long time ago, I took an advanced critical theory course where every week, we had to write a paper analyzing the same text using a different method of critical theory analysis. One week it would me Marxist theory, another week psychoanalytic theory, another week third-work approach (measure it against the standards articulated from another work, like Burke's On the Sublime and Beautiful), another week semiotic and structuralist, another week post-structuralist, another week authorial intent, and so on. The purpose was to show how the same text would produce different meanings depending on the approach used; that instead of focusing on the "correct reading" it was best to think of different theoretical approaches as different tools with which to retrieve meaning. There wasn't a single correct theory- but the theory you used was determinative of the types of meaning you would end up with. A Marxist approach tended to reveal a lot of elements of class struggle and power relations, whereas a psychological analysis is more likely to reveal elements of the characters' conscious and subconscious motivations.
In this way studying literature requires acceptance and even embrace of uncertainty, contradiction, and ambiguity, at least to a certain degree. You can thus learn to say meaningful things about, say, a poem, without driving analysis towards a singular and inevitable end.
The "game design" aspect of rpg theory seems to veer more often towards normative and prescriptive categorization and analysis of types of games, even in its "soft" manifestations ('7 types of gamer' etc). I'm generally more interested in thinking, sometimes abstractly, about rpgs as a social activity (hence my general delight at incoherence, inconsistency, playing "wrong," etc).
1. Role-playing Game Studies. This is an academic work, but is interesting and has the majority of chapters available to the public on-line.
2. The Elusive Shift. Jon Peterson's book. Available at amazon and others.
3. Tabletop RPG Design in Theory and Practice at the Forge, 2001–2012. William J. White. Available at amazon and others (expensive).
4. Designers & Dragons. Shannon Applecline. (I think some of this has been superseded by newer material from Peterson, but sill good). Available with a free TSR section of 100 pages at evilhat.
5. Second Person: Role-Playing and Story in Games and Playable Media. Pat Harrigan. Available at amazon and others.
6. Great roundup of web-based resources at Black & Green Games.
7. Playground Worlds. Some ideas, with a strong emphasis from the Nordic community, available on-line.
Great set of references! To quote from one of the articles in item (1):
Finally, RPG theorizing participates in the politics of culture: To theorize and critically discern RPGs, one must develop both knowledge of and a sense of taste for RPGs. By demonstrating such knowledge and taste in evaluating, appreciating, or rejecting some subject matter, we accrue social and cultural capital (Bourdieu 1984). Every theoretical contribution is thus fundamentally intertwined with one’s social position: to do RPG theory is always also to manage one’s public impression as a theorist within specific (intended) social networks (Goffman 1959). And since RPG theory holds only meager academic and societal status, most RPG theory and criticism is produced by critical amateurs for critical amateurs. “Taste,” as Hennion (2005, 135) puts it, “is a productive activity of critical amateurs”.