• Welcome to this new upgrade of the site. We are now on a totally different software platform. Many things will be different, and bugs are expected. Certain areas (like downloads and reviews) will take longer to import. As always, please use the Meta Forum for site queries or bug reports. Note that we (the mods and admins) are also learning the new software.
  • The RSS feed for the news page has changed. Use this link. The old one displays the forums, not the news.

What would be some good metics to evaluate RPG rules/systems?

Elfcrusher

Explorer
When an interesting sounding RPG comes along, I struggle to get a sense of what the rules are like. Reading reviews I get a good sense for the genre and mood, but what I really want to know is where the rules are in terms of complexity and...well, I'm not really sure what the axes are. I think about the differences between (to grab a few random examples) systems like Dungeon World, various editions of D&D, MERP (!!!), Fantasy Flight Star Wars, and some others and although I have preferences I have trouble specifying how you would measure the differences.

What I'd LOVE is a 2D scatter plot, where games are placed (by consensus?) on two axes. I suspect one axis would be "complexity". What other axis would provide the most useful information? Maybe something about adherence to strict rules, versus narration/interpretation? Although I suspect that axis would correlate pretty strongly to general complexity.

Alternatively (or additionally) having two axes of "chargen complexity/options" and "gameplay complexity" might be interesting.

And then, in addition to numerical ratings, maybe there are some categorical tags that help explain what the rules/systems are like. But, again, I'm not sure what they are.

An analogy of what I'm looking for might be how we shop for cars (size, type, seating, engine, performance, etc.) or houses (price, zip code, style, # of rooms, etc.). Those axes don't give *all* information, but they help you narrow down your search.

What would the equivalent be for RPGs?

Thoughts?
 

Morrus

Well, that was fun
Staff member
It's not like a car. It's much more subjective and much more of an art. You can directly and objectively measure a car's horsepower or weight, but you can't do that with an RPG.

I don't think anything can substitute for actually trying the thing. All the graphs in the world won't give you a sense of how you feel when you play the game.
 

lowkey13

Exterminate all rational thought
An analogy of what I'm looking for might be how we shop for cars (size, type, seating, engine, performance, etc.) or houses (price, zip code, style, # of rooms, etc.). Those axes don't give *all* information, but they help you narrow down your search.

What would the equivalent be for RPGs?

Thoughts?
1. Tables / page.

2. Different dice used (scatterplot of type of dice, e.g., d4, d6, d8, etc. with frequency of use).

3. Frequency of use of percentile dice.

4. Expected reading level (in other words, what is the reading level of the source material).

5. Rules / fluff ratio.

6. Possible number of characters created using base rules (permutations, not including such things as hit point variability, names etc.).

7. Number of ibuprofen required to contemplate the rule in toto.

8. Number of meta-rules (rules to modify other rules).

9. Number of cross-references within text.

10. And .... number of words not found in the OED per page.
 

hawkeyefan

Explorer
The core mechanic may be a good indicator of how the game will play, but I don't think that's something that can be put on an axis in a graph. I agree with Morrus in that it's much more subjective than that.

But there are some things that do help understand how a game will play.....d20 based, d6 based, percentile dice....those give an idea of things. Then there are systems like Powered by the Apocalypse or FATE or Savage Worlds, where there are multiple games, but all share a good deal of mechanics or gameplay design.

But the kind of metrics you're looking for....not sure if there are any suitable ones.
 

ART!

Explorer
boardgamegeek has a pretty good rating system for boardgames. Here's a random sample of their header for a board game:

Screen Shot 2019-06-11 at 11.04.18 AM.png

Their sister site rpggeek does NOT have similar headers for RPGs, but does have the same kind of 1-10 user ratings.

If you do an advanced search for a board game, there are lots of mechanics and other ways to narrow things down. These are screencap samples of the three criteria/categories:

Screen Shot 2019-06-11 at 11.09.59 AM.png Screen Shot 2019-06-11 at 11.10.12 AM.png Screen Shot 2019-06-11 at 11.10.23 AM.png


If you do an advanced search for an rpg, there's much less of that, and more about genre and setting. These are screencap samples of the three criteria/categories:


Screen Shot 2019-06-11 at 11.10.38 AM.png Screen Shot 2019-06-11 at 11.10.46 AM.png Screen Shot 2019-06-11 at 11.11.02 AM.png


I don't have time to draw any conclusions from that.
 
Last edited:

uzirath

Explorer
A major challenge for this endeavor is that one's actual play experience is not always tightly linked to the rules. This is not to say that system doesn't matter — it definitely has an impact — but RPGs are so complicated and have so many aesthetic dimensions that different groups run them entirely differently.

For example, I recall one of the great dungeon master's of my youth would ask everyone to show up with a description of their character on a single 3x5 index card. No stats, no equipment lists, just a bit of backstory fluff and a description of their basic talents. This was 1e AD&D. It sure didn't feel like a lot of other AD&D tables. Similarly, I've played D&D games in other editions that were all combat and others that were mostly role-playing with precious few die rolls in any given session. I've played at GURPS tables where every optional rule seemed to be in effect with scores of books available for consultation and where the tactical combat system was really its own hyper-complex battle simulation. And I've played at GURPS tables that were minimalist with simple wildcard skills, zero combat, and no books at all.

Again, this isn't to say that the rules systems themselves might not be measurable in some manner, and compared against each other, but I wonder how much utility this would have in terms of predicting the actual experience of play? A narrative review may have more utility because it won't be constrained by the elements that are being measured, but reviewers may need to do a better job at describing the crunch
 

ART!

Explorer
You could have users rate a game overall on some scale, and/or rate things like layout, indexing, clarity, organization, etc.

You could list if there's randomization, and if so what is used for randomization (generic 52-card deck, proprietary cards, dice - of what kind and number, etc...). Then you could list general mechanics like "roll under", "roll over", "count successes", "degrees of success", "exploding dice", etc.
 
Last edited:

Umbran

Mod Squad
Staff member
Don't ask, "What would be a good metric?"

Ask instead: "What matters to you about a game?" Not just as a thought exercise - Make an actual list of things that matter to you.

We can try to separate what of those things are GM/group dependent, and what's rules-dependent, and what measures we could put that could be useful.

Metrics should be chosen to answer specific questions.
 

Beleriphon

Totally Awesome Pirate Brain
Metrics should be chosen to answer specific questions.
I think there are a few good metrics for rules that are universal. Organization, clarity of writing, really anything one would use to evaluate whether any reference book is good at doing what it is supposed to do.
 

Elfcrusher

Explorer
It's not like a car. It's much more subjective and much more of an art. You can directly and objectively measure a car's horsepower or weight, but you can't do that with an RPG.

I probably shouldn't have used the word "metric" because it suggests something objective, whereas I'm imagining a value that's subjective, and/or maybe an aggregate/consensus value.

But I do think think there are things that can be evaluated this way, such as the examples I gave.

Or maybe it's sort of a correspondence analysis, where you would say the game in question has these features like games A, B, and C, and these other features like games D, and E, but this other feature like game F.


I don't think anything can substitute for actually trying the thing. All the graphs in the world won't give you a sense of how you feel when you play the game.
The same could be said for driving cars and visiting houses. And yet time is finite.
 

Elfcrusher

Explorer
I think there are a few good metrics for rules that are universal. Organization, clarity of writing, really anything one would use to evaluate whether any reference book is good at doing what it is supposed to do.
Mmm, I don't really care about...or at least I wasn't referring to...organization, clarity of writing, index.

I've backed games on Kickstarter because the description was awesome, or the genre is something that appeals to me, or maybe just because the artwork was great. Then I get the game and I think, "Ugh...this reminds me of (insert game I hate)." And it's not about production quality, it's about the underlying system.

And on the flip side, I wonder what games I would love but I'm missing out on because something...the artwork?...didn't appeal to me.
 

Elfcrusher

Explorer
1. Tables / page.

2. Different dice used (scatterplot of type of dice, e.g., d4, d6, d8, etc. with frequency of use).

3. Frequency of use of percentile dice.

4. Expected reading level (in other words, what is the reading level of the source material).

5. Rules / fluff ratio.

6. Possible number of characters created using base rules (permutations, not including such things as hit point variability, names etc.).

7. Number of ibuprofen required to contemplate the rule in toto.

8. Number of meta-rules (rules to modify other rules).

9. Number of cross-references within text.

10. And .... number of words not found in the OED per page.
Funny...but possible character permutations is on the right track of...something. At least, I would find that an interesting number. Better yet, a tree-like diagram showing the permutations. The shape of the tree would actually tell you a lot about the game. Are choices front-loaded, or do they appear evenly over many levels? Are choices restricted based on early decisions, or is it (yet another pathetic attempt at) a pure classless a-la-carte system? Etc.
 

Beleriphon

Totally Awesome Pirate Brain
Perhaps, but if I said a game was organized and written like the AD&D Dungeon Master's Guide, that tells you quite a bit about the game, without getting into the specificities of the rules. For example you might enjoy that book, and find it worthwhile read, while I on the other hand am constantly boggled by the sheer loquaciousness that Gygax felt was necessary.

Presentation matters, and its something that can affect anybody's willingness to just read a book, never mind understand the content.
 

Umbran

Mod Squad
Staff member
I think there are a few good metrics for rules that are universal.
Well, there may be some thigns that nearly everyone cares about - some specific questions everyone wants to know the answer to. That's okay.

But you should approach it *from the question* rather than from the answer.

Organization, clarity of writing, really anything one would use to evaluate whether any reference book is good at doing what it is supposed to do.
Well, where I come from, the defining characteristic of a thing we call a "metric" is that it is *measurable*, preferably in as objective a manner as possible.

I don't know how to measure clarity of writing. You can get user ratings on just about anything you want, but darn does that have pitfalls.
 

Umbran

Mod Squad
Staff member
I probably shouldn't have used the word "metric" because it suggests something objective, whereas I'm imagining a value that's subjective, and/or maybe an aggregate/consensus value.
That's fair.

And then what we are looking for is a list of things for people to rate a game on. There's an arti and science of that, too, which can get pretty deep.

There are a few things that can get hairy when you are taking user ratings. Things like "complexity", where you might wonder if everyone has the same concept fo what you are talking about. But, you can sometimes get at what you mean without using particular terms. A proxy for one form of complexity might be, "ease of use in play," or, "time to complete a typical combat encounter", or "typical number of modifiers to a die roll" - which are actually quite indicative of things people want to know at the table.
 

Tony Vargas

Adventurer
1. Tables / page.

2. Different dice used (scatterplot of type of dice, e.g., d4, d6, d8, etc. with frequency of use).

3. Frequency of use of percentile dice.

4. Expected reading level (in other words, what is the reading level of the source material).

5. Rules / fluff ratio.

6. Possible number of characters created using base rules (permutations, not including such things as hit point variability, names etc.).

7. Number of ibuprofen required to contemplate the rule in toto.

8. Number of meta-rules (rules to modify other rules).

9. Number of cross-references within text.

10. And .... number of words not found in the OED per page.

You've got a lot of 1e books and a sense of humor, how bout compiling those stats?
 
I share your feeling of disappointment in finding out a game is not at all what I thought it was or was looking for. I’ve begun to believe that we really need some well-defined subcategories for this poorly-defined misnomer we call “role-playing games”. I’ve seen discussions where people will respond to questions like “what game should I buy for <genre>?” with gushing recommendations for a variety of games that differ drastically from each other in fundamental ways—so much so that it’s quite likely the buyer will be highly disappointed with some of them if they have any expectations about what a <genre> RPG looks like, and aren’t just in a mindset of looking to try something new, don’t much care what, run with <genre> to spark ideas for suggestions. (I apologize for that sentence; it was atrocious even by my own low standards.) If, on the other hand, the buyer could have asked with a set of parameters that were generally understood, half of those suggestions never would have come up (or would have come up with an express statement that they didn’t quite fit—which can actually be really useful for providing new options without being confusing), and the potential buyer would get the information they need and not be stuck with a pretty game they’re never going to play.

So: What you are looking for is doable. I don’t know how to do it yet; but it’s doable.

I have some brainstorms pointing in that direction. I’ll divide them into part 1 and part 2. I do this because part 1 is short, but might garner a lot of discussion, and that’s not what I’m really intending since part 2 is more directly addressing this particular thread. So I’m dividing them up by intent.

Part 1: A very general categorization like G/N/S theory would be useful here. I don’t hold to all of the developments from that theory, but I do like the basic initial insights and have yet to develop any other alternative taxonomy that is better. While it is imperfect it, quite frankly, works. If someone tells me that a game is “highly narrativist”, or “heavy on simulationism”—despite the fact that the theory says you can’t classify games themselves that way—I get some really valuable information about that system before I even look at it. And so far I have never derived mistaken impression from such simplistic statements as saying where a game fits into those styles. I’m very open to a better system (and always trying to come up with one), but GNS works at a basic level.

How would you use this sort of thing for classifying games? I’d go with identifying intent and implementation separately. For instance, you might say that a game’s intent is highly narrativist, but the implementation is moderately simulationist with some awkwardly bolted on narrativist flourishes. Or you might say another game is an unusually successful blend of a baseline of simulationist task resolution, with gamist play elements, and integrated narrativism throughout (how I might described Torg: Eternity). Tell me that about any game and you’ve given me enough to go into it with a significantly lessened chance of having completely missed my expectations.

Part 2: I think we can do better by getting into more detail about various elements of a game. Sufficient detail can make it a lot easier to identify objective information that can be conveyed. It’s up to individuals to say whether they think it was done well or not, but we should be able to objectively define much of what was actually done. A few ideas.

Rules Heaviness: One could separate this into a couple of categories, such as quantity of crunch, and diversification of systems. So a game that only has a few basic systems, but has books and books of additional crunch content (say, books with additional powers, races, classes, that sort of thing) that uses those same systems would be high crunch quantity but low system diversification. The opposite would be a game whose rules all fit within 30 pages, but has a different subsystem for everything. And you could have rules that are heavy on both of those or light on both of those, etc. That’s much more useful than saying a system it complex or rules heavy.

Play-Style Intention: A couple of the ones that I find most prevalent are Plot Development—where you are really focused first and foremost on telling a story—and Setting Exploration—where the game is designed with a really interesting world for you to sandbox around in and do whatever. I suspect these are somewhat opposed to one another—the more you are just soaking in the world the less you can be focusing your play experience on a story, and vice versa. But I’m not sure that’s the case. In any event, those are some things where the design intent is usually pretty obvious. A game isn’t limited to just one intent either. “Works equally well in providing a sandbox to explore as it does in providing a strong story-focus” is a perfectly good description. There are other elements that need to be addressed here. I don’t have a comprehensive list, though I’m sure it could be narrowed down to a few big categories that most people will find useful.

Task Resolution: Others have already addressed this, and this is one of the ones that tends to be put front and center before you even try the system, though sometimes it isn’t. Good and concise explanations here would be useful. “Uses d10s” says nothing. I’m not sure what sort of descriptive conventions would work here, but there is definitely room for improvement since the concise descriptions that work aren’t accessible unless you already have a lot of experience with RPGs and enough experience comparing them to get the jargon the writer chose to employ, while more accessible statements tend to be uninformative.

Setting: This is one of the easiest ones. Modern Fantasy. Space Opera. Wild West with Zombies and Voodoo. What’s needed here is a simple, one sentence, description that tells you what distinguishes this Space Opera from all of the others in such a way that the reader immediately knows if they will like it or not.

Scope: This expands on setting by explaining a few things you might not get in that first sentence. For instance, is meta-plot a thing? How much of a thing it is? Does the product line have an in-world timeline that marches along with mega-adventures and novels progressing it, or is there a vague menace out there somewhere with hints showing up in setting products that are otherwise more or less time-agnostic or snapshots of parts of the setting? Is there no meta-plot at all, and just an initial setup and then you do what you want? Is this system a toolbox for making your own setting, and if so what are the limits of what sort of settings it supports? Scope could also go beyond setting and include rules. Are there things that you might expect the rules to cover but that they don’t? What sort of power levels are supported? Are there any limitations in what levels of power player character have access to compared to other characters in the world? If so, are these inherent to the premise (the Darkened Evilites gain great power by foul deeds), or are they arbitrary? Here’s one that you don’t see brought up much: Is there an assumption that players start at a specific (generally low) power level (ie, “Level 1”)? Personally I think that’s a ridiculous assumption that continues to be built into games, but most people just assume it’s the case, so if a game doesn’t do that, it’s a feature worth pointing out.

Product Investment: How much do you get in the core books? How many other books are there, or are there expected to be, and what are you getting out of them? Do you just get a taste of certain elements in the core books and need further books to really get the full material on those? (Here it is definitely worth getting reviewer opinions, because publishers have a vested interest that might bias their descriptions, ie “all you need to….!” doesn’t necessarily mean, “all I actually want to…”) Tying-in to scope, how much investment do you need to get the scope you want? If it’s space opera, does the core material focus in on a certain part of the galaxy, and you need supplements to get the rest—or does it give you a high-level overview of everywhere and supplements go into more depth? How many supplements are there, and how much material do you get out of each?

Game Presentation: This is sort of a catchall category for things that are more or less external to the system and setting, but will matter to consumers and can be categorized. One might be maturity rating (both as far as what sort of players it might appeal to, as well as how much mature content it contains—which are by no means correlatable). Another would be the amount and quality of art and how enjoyable the books are to read by themselves. Is it something that sparks the imagination with lots of ideas (whether for adventures to run with this game, or just to get you thinking in general)? Is it concisely organized?

For time purposes I’m going to stop there. I see that I’ve wandered from classifications to something more like review advice, but I think most of what I talked about (and whatever I missed) could be condensed down into maybe 10 different categories. They wouldn’t all be ratable on a numeric scale, but they could be rated with a few words each, ie “highly focused on setting exploration, minimal emphasis on storylines”, usually in a similar format. Some categories would be more akin to crates you stick some keywords into. You might throw this into a Scope category for a game: Meta-plot heavy, solid world-overview, setting details in supplements, zero-to superhero advancement.
 

Tony Vargas

Adventurer
.
I don't know how to measure clarity of writing.
Google it

It's not like "I know it when I see it," it's a thing.

And, yes, Clarity is a major plus for any rule system that's not trying to evoke the DM-centric feel of classic D&D. It cuts down on the need for rulings, makes the game easier to understand, run, and play, and cuts down on the kinds of endless arguments that ruin sessions, but keep internet discussion forums thriving...

...er, on deeper cogitation clarity is surely an exemplar of those great philosophical imponderables, no game should be expected to, or could be said to manifest a supernumary surfeit of, nor contrarywise, to evince.an alarming paucity of clarity on anything but a wholly subjective basis.
 

Elfcrusher

Explorer
It cuts down on the need for rulings...
See, I actually think that "rulings" are...or can be...a good thing, and it's one of the things I'd like to know about a game. To what extent do the rules attempt to specify outcomes for every scenario, versus how much is left to interpretation? You can have clear writing that leaves some questions open, if that's the intent.
 

Dannyalcatraz

Moderator
Staff member
Don't ask, "What would be a good metric?"

Ask instead: "What matters to you about a game?" Not just as a thought exercise - Make an actual list of things that matter to you.
It may just be me, but what matters to me depends on all kinds of things. Worse yet, those things can change depending on other things.

...which is why I’ve invested so much $$$ in so many different RPG systems.

...and why- even though HERO is my favorite system, and I feel I can model whatever I want with it- I prefer other systems for other kinds of games. I mean I have actually done Fantasy HERO campaigns that simultaneously modeled playable versions of character classes from multiple editions of D&D and it ran just fine. But I’d rather play a D&D campaign in a D&D system.

And I wouldn’t even dream of trying to run a Fantasy Trip/In the Labyrinth game In Fantasy HERO because that would kind of miss the point of playing TFT/ITL in the first place.
 

Advertisement

Top