What would be some good metics to evaluate RPG rules/systems?

Riley37

Visitor
Some commenters have raised analogies to summarized information about cars and houses. People generally buy one car at a time, out of many options, and one house at a time, out of several or many options. So is this proposed system also meant to inform purchase choices?

A system for comparing TRPGs might be useful for informing choices about registering for convention games. I might prefer a game along these lines: "roll 1d6 if you're in unfavorable situations, 2d6 for average tasks, 3d6 if you're well-trained, well-equipped, or in favorable circumstances." I might dislike games along these lines: "roll 1d100 for accuracy, applying modifiers for characteristics, skill ranks, equipment category, and other factors; then roll 3d6 for power, also with modifiers; compare your Accuracy outcomes to the target or task's Elusive rating (also rolled on d100) and your Power outcome to the target or task's Resistance rating (a fixed number from 0 to 25). If I can sign up for either the 10AM Saturday game of Hackmaster, or the 10AM Saturday game of Doctor Who, then I might enjoy knowing which game has more moving parts in its action resolution system.

Yes, metrics should have actual numbers. Sometimes, failing hard numbers, rankings are useful. I could not tell you the numerical Scoville ratings of the hot, medium, mild and supernova salsas at my local tacqueria; but I can accurately rank them from most to least spicy. (That ranking isn't the alphabetical order I used, when listing the categories, just now.)

Metics, on another hand, were foreigners living in an ancient Greek city who had some of the privileges of citizenship.
 

Bedrockgames

Adventurer
When an interesting sounding RPG comes along, I struggle to get a sense of what the rules are like. Reading reviews I get a good sense for the genre and mood, but what I really want to know is where the rules are in terms of complexity and...well, I'm not really sure what the axes are. I think about the differences between (to grab a few random examples) systems like Dungeon World, various editions of D&D, MERP (!!!), Fantasy Flight Star Wars, and some others and although I have preferences I have trouble specifying how you would measure the differences.

What I'd LOVE is a 2D scatter plot, where games are placed (by consensus?) on two axes. I suspect one axis would be "complexity". What other axis would provide the most useful information? Maybe something about adherence to strict rules, versus narration/interpretation? Although I suspect that axis would correlate pretty strongly to general complexity.

Alternatively (or additionally) having two axes of "chargen complexity/options" and "gameplay complexity" might be interesting.

And then, in addition to numerical ratings, maybe there are some categorical tags that help explain what the rules/systems are like. But, again, I'm not sure what they are.

An analogy of what I'm looking for might be how we shop for cars (size, type, seating, engine, performance, etc.) or houses (price, zip code, style, # of rooms, etc.). Those axes don't give *all* information, but they help you narrow down your search.

What would the equivalent be for RPGs?

Thoughts?
If you are looking at cars as an analogy, I think basic information on a table so you can compare systems and settings easy (System, Cost, Complexity, Style//Genre, Rating/Ranking (if applicable), etc).
 
See, I actually think that "rulings" are...or can be...a good thing, and it's one of the things I'd like to know about a game. To what extent do the rules attempt to specify outcomes for every scenario, versus how much is left to interpretation?
That's not a matter of clarity, but of scope or completeness, and not an interpretation or a ruling, but simply adding - formally or informally - to the game.

For instance, if a game includes no prices, stats, or rules for weapons, then armed combat might be outside its scope - maybe it's all about boxing, IDK. The rules it does present might be clear enough that no rulings or interpretations are called for, everyone who reads the rules can play the game without confusion or argument over what the rules mean. But, when a player decides his character will hit someone with a folding chair, or try to buy a gun, the GM will either deny him, or add to the game.

Now, if that game was actually supposed to be a film noir detective RPG, the lack of any rules covering firearms would be more a matter of the rules being incomplete, than of them being unclear.
 

Elfcrusher

Adventurer
For instance, if a game includes no prices, stats, or rules for weapons, then armed combat might be outside its scope -
I know you said "might", but...Dungeon World?

Sure, some weapons have stats, but there's no list. And definitely no prices. You do damage based on your class, regardless of how you decide to describe your weapon. Dual-wielding rolling pins? Same damage.

That's the kind of thing I'd like to know about in evaluating a game.

Again, maybe the solution is not to quantify things, but to rate games on a bunch of things in terms of other games they are like?
 
I know you said "might", but...Dungeon World?

Sure, some weapons have stats, but there's no list. And definitely no prices. You do damage based on your class, regardless of how you decide to describe your weapon. Dual-wielding rolling pins? Same damage.
That's still rules for using weapons, just less detailed ones.

And it's hard to say a game is X level of detailed. For instance, 1e AD&D had a more detailed list of weapons, with more detailed stats than 3e, but 3e had more detailed rules for using them.

How do you rate those differences?

Again, maybe the solution is not to quantify things, but to rate games on a bunch of things in terms of other games they are like?
What if a game were to (gasp) do something differently than other games?

In the 80s, 'Core Systems' seemed to me a big part of the industry paradigm. The idea being, if you liked, say, CoC, you'd easily be able to play other BRP based games. The culmination was probably GURPS.

Point being, that's kinda what you're looking for: new games that are enough like existing games you're already familiar with and like.
 

Celebrim

Legend
I work in software development.

There is a general sense in the industry, that for two given methods for solving a problem, the one that is lower complexity tends to be the better solution. We are certainly encouraged to right rules that reduce complexity according to very similar measures as some have proposed regarding RPG rules - length of the rules, readability, number of branching paths, number of subsystems, depth of hierarchies, number of cross references, number of meta-rules, etc. The trouble is that it turns out that measuring any of those things in a meaningful manner is really hard, and often two metrics are in tension with each other - for example, you can make the rules more compact, but only at the cost of reducing readability, increasing cross references, and/or increasing the number of meta-rules. So then there is the problem of how you would weight the tradeoff. And then it turns out that things like space, complexity, and processing time are also trade offs, so that you can increase "speed of play" but only at the cost of longer and more complex rules, such as increasing the number of tables.

So in short, if you could solve this problem in an academic sphere with relatively little economic impact, then you could solve this problem in an academic sphere with enormous economic impact, and probably set yourself up for a very prestigious prize in mathematics or economics. It would be certainly the sort of thing that could get you nominated for a Nobel prize or the like.
 

Elfcrusher

Adventurer
Point being, that's kinda what you're looking for: new games that are enough like existing games you're already familiar with and like.
That's quite a leap, and not even a remotely accurate conclusion. I asked the question whether rating games on their similarity (on various dimensions) to existing games would be a useful thing. I didn't say that I'm looking for new games that are like existing ones.

In fact, in some cases knowing that new game X is really far away from existing game Y, on nearly all dimensions, would be a positive sign for me.

That said, you raise a good point: if a game is totally unlike existing games, how do you evaluate it using that method.
 
Last edited:

Elfcrusher

Adventurer
I work in software development.

There is a general sense in the industry, that for two given methods for solving a problem, the one that is lower complexity tends to be the better solution. We are certainly encouraged to right rules that reduce complexity according to very similar measures as some have proposed regarding RPG rules - length of the rules, readability, number of branching paths, number of subsystems, depth of hierarchies, number of cross references, number of meta-rules, etc. The trouble is that it turns out that measuring any of those things in a meaningful manner is really hard, and often two metrics are in tension with each other - for example, you can make the rules more compact, but only at the cost of reducing readability, increasing cross references, and/or increasing the number of meta-rules. So then there is the problem of how you would weight the tradeoff. And then it turns out that things like space, complexity, and processing time are also trade offs, so that you can increase "speed of play" but only at the cost of longer and more complex rules, such as increasing the number of tables.

So in short, if you could solve this problem in an academic sphere with relatively little economic impact, then you could solve this problem in an academic sphere with enormous economic impact, and probably set yourself up for a very prestigious prize in mathematics or economics. It would be certainly the sort of thing that could get you nominated for a Nobel prize or the like.
Again, though, I'm not looking for an objective metric. 10 software engineers with sufficient experience and knowledge would be likely to give a given piece of source code a somewhat similar rating for compactness and elegance. Exactly the same value? No. But in the same ballpark.

Likewise, if 15 reviewers give Chargen Complexity an average of 1.7 out of 5, that's good enough for me.

Anyway, I'm not sure the idea I'm proposing has any merit, but it's something I've wondered about.
 

Satyrn

Visitor
10. And .... number of words not found in the OED per page.
Objection!

This metric would wildly skew the position of my fantasy heartbreaker, inspired by nostalgia for my time gaming in Montreal with a Quebecois group even though the only French I knew was what I learned in school, not really paying attention in French class.

Les Dungeonnes et Les Dragóns, written in what I lovingly dub Satyrninian Canadian is full of words found in neither the OED, nor whatever dictionary the French use.
 
I didn't say that I'm looking for new games that are like existing ones.
You didn't say it, may not believe it, but the rating systems you're thinking about - and your reactions to what's been proposed as possible ratings - make me think that's what you're looking for...
... or, at least, what you'll end up finding, by looking in the manner that you seem to be leaning towards.

That said, you raise a good point: if a game is totally unlike existing games, how do you evaluate it using that method.
You'd end up evaluating it negatively (in the sense of judging it to be bad, or in the sense of only being able to define what it's not), or even as not being a game at all.
 

Celebrim

Legend
Again, though, I'm not looking for an objective metric. 10 software engineers with sufficient experience and knowledge would be likely to give a given piece of source code a somewhat similar rating for compactness and elegance. Exactly the same value? No. But in the same ballpark.
You are possibly right. Certainly, 10 software engineers with sufficient experience and knowledge would be able to agree as to when code was badly written.

But one of the underlying assumptions of your statement is that they software engineers were reasonably familiar with the language paradigm of the code. I honestly don't have have a very good feel for what very elegant Lisp or Prolog code would look like, because while I've been exposed to toy solutions in those languages (and asked to write a few toy solutions), I've never seen anything resembling a real solution in those languages and the paradigm that they use is radically different. Likewise, on first appearance, assembly language looks like gobbly-gook to me, and I'd have no feel for how elegant it was nor am I certain that even good assembly language programmers would see the structure of the program quickly.

The problem is that English or any other natural language as a communication paradigm more resembles Lisp or Assembly Language in terms of ability to grasp it's elegance than say C++ does a dozen programmers who all share 20 years experience with the language. Even though I use English all the time, as a means of expressing rules, the subtleties of the problem or the elegance of the solution are often much easier to miss than defects in a section of code. Nor is it obvious that any 10 people highly expressive in English will agree as to what constitutes good writing in English.

Anyway, I'm not sure the idea I'm proposing has any merit, but it's something I've wondered about.
It definitely has merit. We just don't understand the subject well enough to answer your question. In fact, my suspicion is that the answer is so complicated that a mere human mind will not be able to grasp it. We'll likely have to build a mind subtle and powerful enough to get the answer, and then build another one to try to explain it to us in language simple enough for us to grasp.
 

dragoner

Dying in Chargen
I think it could work totally fine, user rating, genre, system, complexity, and price. These things have a tendency to evolve, that is good too.
 

Riley37

Visitor
Nor is it obvious that any 10 people highly expressive in English will agree as to what constitutes good writing in English.
The range of what ten people consider "literary" English has arisen on another thread. Arisen, or perhaps descended. At least I've learned a new phrase: "high Gygaxian".

I have my doubts about whatever design team developed the English language. I hope it was delivered on time, for a low cost, because it sure isn't winning any prizes for ease of use, nor for internal documentation.
 

pemerton

Legend
That's not a matter of clarity, but of scope or completeness, and not an interpretation or a ruling, but simply adding - formally or informally - to the game.

For instance, if a game includes no prices, stats, or rules for weapons, then armed combat might be outside its scope - maybe it's all about boxing, IDK. The rules it does present might be clear enough that no rulings or interpretations are called for, everyone who reads the rules can play the game without confusion or argument over what the rules mean. But, when a player decides his character will hit someone with a folding chair, or try to buy a gun, the GM will either deny him, or add to the game.
This post makes many assumptions about how a game might work. Many games don't require "adding to the game" (eg by way of new subsystems, or new modifiers, or whatever) because they have resolution systems that are relatively straightforward to extrapolate to novel situations.

I appreciate that D&D, historically, has not been such a system - it emphasises particular subsystems rather than general resolution - and many other games are similar in this respect. But that's not the only design path. (Thinking of some early post-D&D games, Classic Traveller is sub-system based, but Tunnels & Trolls mostly isn't.)

And I think it's quite misleading - as in, for instance, won't give someone like [MENTION=6801328]Elfcrusher[/MENTION] the information being looked for about a new game - just to say that a universal or extrapolation-based resolution system is just like a sub-system based game but "more detailed".
 

Elfcrusher

Adventurer
I thought of a (subjective) rating I'd like: how easy it is to mix new and veteran characters on the same adventure. I think that says a lot about game design. In my search panel I'd like to be able to filter for games where "low level characters can meaningfully contribute to adventures with high level characters."

EDIT: And others might prefer the opposite; they might specifically want more Zero2Hero because it feels more epic.
 
Last edited:

Riley37

Visitor
Perhaps someone designing this system could produce multiple versions. Version A asks six questions. Version B asks six questions, three of which overlap with Version A.

Issue both versions. Have people rate games. See if those ratings are useful to others. Ask those who read the ratings: if your first awareness of a game was its rating, and then you played the game, did your experience of the game match the rating, that is, do you think, afterwards, that the rating was accurate?

Assess results. A good result: "I read the ratings on Game X, and then played Game X. The ratings helped me anticipate what I did and didn't like about Game X."
A bad result: "The ratings gave me no useful information about Game X."
Another bad result: "Based on ratings, I tried Game X. I think the ratings misinformed my decision and I regret taking the ratings seriously."

Compare how often Version A yields good results, versus how often Version B yields good results.

Also: account for cognitive bias, across each step of this process. For example, people who *expect* to enjoy a game, might be more likely to report enjoyment of that game, than people who go in with no pre-conceptions.

Come to think of it, information more specific than ratings, might also be useful. "You won't enjoy FooBarQuest" might be useful, if I'm choosing games at a convention and I can either try to get into a FooBarQuest game or try to get into a Pathfinder game.

When I want to try a new game, and FooBarQuest is the only option aside from games I've already played, the situationally useful advice is "If you play FooBarQuest, then play a martial class, because the magic rules are a mess; and don't play a quapatir, they look awesome but you'll never get to use the abilities." At which point, maybe what I want is a review?
 

Elfcrusher

Adventurer
Perhaps someone designing this system could produce multiple versions. Version A asks six questions. Version B asks six questions, three of which overlap with Version A.

Issue both versions. Have people rate games. See if those ratings are useful to others. Ask those who read the ratings: if your first awareness of a game was its rating, and then you played the game, did your experience of the game match the rating, that is, do you think, afterwards, that the rating was accurate?

Assess results. A good result: "I read the ratings on Game X, and then played Game X. The ratings helped me anticipate what I did and didn't like about Game X."
A bad result: "The ratings gave me no useful information about Game X."
Another bad result: "Based on ratings, I tried Game X. I think the ratings misinformed my decision and I regret taking the ratings seriously."

Compare how often Version A yields good results, versus how often Version B yields good results.

Also: account for cognitive bias, across each step of this process. For example, people who *expect* to enjoy a game, might be more likely to report enjoyment of that game, than people who go in with no pre-conceptions.

Come to think of it, information more specific than ratings, might also be useful. "You won't enjoy FooBarQuest" might be useful, if I'm choosing games at a convention and I can either try to get into a FooBarQuest game or try to get into a Pathfinder game.

When I want to try a new game, and FooBarQuest is the only option aside from games I've already played, the situationally useful advice is "If you play FooBarQuest, then play a martial class, because the magic rules are a mess; and don't play a quapatir, they look awesome but you'll never get to use the abilities." At which point, maybe what I want is a review?
I'm wary of the word "ratings" (even though I started using it instead of "metric") because I feel like there's an implicit "higher is better" assumption, and what I want to know is where games lie on various spectra. I'm really not trying to figure out if a game is good, I want to know what it's lik, in a searchable/filterable way. (That is, without spending countless hours combing through reviews, most of which in my experience don't tell me what I want to know anyway.)
 

Riley37

Visitor
So noted. A game which rates higher on "complexity" isn't necessarily more "the game you want" than a game which rates lower; if you're looking for a game to run with children, the reverse. I'll adjust my usage to metrics.

(I will only rate a game on metics, if it's set in ancient Athens.)
 

Advertisement

Top