What would be some good metics to evaluate RPG rules/systems?

Tony Vargas said:

... or, at least, what you'll end up finding,

Click to expand...

Now that might be true.

Elfcrusher said:

Again, though, I'm not looking for an objective metric. 10 software engineers with sufficient experience and knowledge would be likely to give a given piece of source code a somewhat similar rating for compactness and elegance. Exactly the same value? No. But in the same ballpark.

Click to expand...

You are possibly right. Certainly, 10 software engineers with sufficient experience and knowledge would be able to agree as to when code was badly written.

But one of the underlying assumptions of your statement is that they software engineers were reasonably familiar with the language paradigm of the code. I honestly don't have have a very good feel for what very elegant Lisp or Prolog code would look like, because while I've been exposed to toy solutions in those languages (and asked to write a few toy solutions), I've never seen anything resembling a real solution in those languages and the paradigm that they use is radically different. Likewise, on first appearance, assembly language looks like gobbly-gook to me, and I'd have no feel for how elegant it was nor am I certain that even good assembly language programmers would see the structure of the program quickly.

The problem is that English or any other natural language as a communication paradigm more resembles Lisp or Assembly Language in terms of ability to grasp it's elegance than say C++ does a dozen programmers who all share 20 years experience with the language. Even though I use English all the time, as a means of expressing rules, the subtleties of the problem or the elegance of the solution are often much easier to miss than defects in a section of code. Nor is it obvious that any 10 people highly expressive in English will agree as to what constitutes good writing in English.

Anyway, I'm not sure the idea I'm proposing has any merit, but it's something I've wondered about.

Click to expand...

It definitely has merit. We just don't understand the subject well enough to answer your question. In fact, my suspicion is that the answer is so complicated that a mere human mind will not be able to grasp it. We'll likely have to build a mind subtle and powerful enough to get the answer, and then build another one to try to explain it to us in language simple enough for us to grasp.

Celebrim said:

We'll likely have to build a mind subtle and powerful enough to get the answer....

Click to expand...

We could call it "Deep Thought."

I think it could work totally fine, user rating, genre, system, complexity, and price. These things have a tendency to evolve, that is good too.

Celebrim said:

Nor is it obvious that any 10 people highly expressive in English will agree as to what constitutes good writing in English.

Click to expand...

The range of what ten people consider "literary" English has arisen on another thread. Arisen, or perhaps descended. At least I've learned a new phrase: "high Gygaxian".

I have my doubts about whatever design team developed the English language. I hope it was delivered on time, for a low cost, because it sure isn't winning any prizes for ease of use, nor for internal documentation.

Tony Vargas said:

That's not a matter of clarity, but of scope or completeness, and not an interpretation or a ruling, but simply adding - formally or informally - to the game.

For instance, if a game includes no prices, stats, or rules for weapons, then armed combat might be outside its scope - maybe it's all about boxing, IDK. The rules it does present might be clear enough that no rulings or interpretations are called for, everyone who reads the rules can play the game without confusion or argument over what the rules mean. But, when a player decides his character will hit someone with a folding chair, or try to buy a gun, the GM will either deny him, or add to the game.

Click to expand...

This post makes many assumptions about how a game might work. Many games don't require "adding to the game" (eg by way of new subsystems, or new modifiers, or whatever) because they have resolution systems that are relatively straightforward to extrapolate to novel situations.

I appreciate that D&D, historically, has not been such a system - it emphasises particular subsystems rather than general resolution - and many other games are similar in this respect. But that's not the only design path. (Thinking of some early post-D&D games, Classic Traveller is sub-system based, but Tunnels & Trolls mostly isn't.)

And I think it's quite misleading - as in, for instance, won't give someone like [MENTION=6801328]Elfcrusher[/MENTION] the information being looked for about a new game - just to say that a universal or extrapolation-based resolution system is just like a sub-system based game but "more detailed".

I thought of a (subjective) rating I'd like: how easy it is to mix new and veteran characters on the same adventure. I think that says a lot about game design. In my search panel I'd like to be able to filter for games where "low level characters can meaningfully contribute to adventures with high level characters."

EDIT: And others might prefer the opposite; they might specifically want more Zero2Hero because it feels more epic.

Perhaps someone designing this system could produce multiple versions. Version A asks six questions. Version B asks six questions, three of which overlap with Version A.

Issue both versions. Have people rate games. See if those ratings are useful to others. Ask those who read the ratings: if your first awareness of a game was its rating, and then you played the game, did your experience of the game match the rating, that is, do you think, afterwards, that the rating was accurate?

Assess results. A good result: "I read the ratings on Game X, and then played Game X. The ratings helped me anticipate what I did and didn't like about Game X."
A bad result: "The ratings gave me no useful information about Game X."
Another bad result: "Based on ratings, I tried Game X. I think the ratings misinformed my decision and I regret taking the ratings seriously."

Compare how often Version A yields good results, versus how often Version B yields good results.

Also: account for cognitive bias, across each step of this process. For example, people who *expect* to enjoy a game, might be more likely to report enjoyment of that game, than people who go in with no pre-conceptions.

Come to think of it, information more specific than ratings, might also be useful. "You won't enjoy FooBarQuest" might be useful, if I'm choosing games at a convention and I can either try to get into a FooBarQuest game or try to get into a Pathfinder game.

When I want to try a new game, and FooBarQuest is the only option aside from games I've already played, the situationally useful advice is "If you play FooBarQuest, then play a martial class, because the magic rules are a mess; and don't play a quapatir, they look awesome but you'll never get to use the abilities." At which point, maybe what I want is a review?

Riley37 said:

Perhaps someone designing this system could produce multiple versions. Version A asks six questions. Version B asks six questions, three of which overlap with Version A.

Issue both versions. Have people rate games. See if those ratings are useful to others. Ask those who read the ratings: if your first awareness of a game was its rating, and then you played the game, did your experience of the game match the rating, that is, do you think, afterwards, that the rating was accurate?

Assess results. A good result: "I read the ratings on Game X, and then played Game X. The ratings helped me anticipate what I did and didn't like about Game X."
A bad result: "The ratings gave me no useful information about Game X."
Another bad result: "Based on ratings, I tried Game X. I think the ratings misinformed my decision and I regret taking the ratings seriously."

Compare how often Version A yields good results, versus how often Version B yields good results.

Also: account for cognitive bias, across each step of this process. For example, people who *expect* to enjoy a game, might be more likely to report enjoyment of that game, than people who go in with no pre-conceptions.

Come to think of it, information more specific than ratings, might also be useful. "You won't enjoy FooBarQuest" might be useful, if I'm choosing games at a convention and I can either try to get into a FooBarQuest game or try to get into a Pathfinder game.

When I want to try a new game, and FooBarQuest is the only option aside from games I've already played, the situationally useful advice is "If you play FooBarQuest, then play a martial class, because the magic rules are a mess; and don't play a quapatir, they look awesome but you'll never get to use the abilities." At which point, maybe what I want is a review?

Click to expand...

I'm wary of the word "ratings" (even though I started using it instead of "metric") because I feel like there's an implicit "higher is better" assumption, and what I want to know is where games lie on various spectra. I'm really not trying to figure out if a game is good, I want to know what it's lik, in a searchable/filterable way. (That is, without spending countless hours combing through reviews, most of which in my experience don't tell me what I want to know anyway.)

So noted. A game which rates higher on "complexity" isn't necessarily more "the game you want" than a game which rates lower; if you're looking for a game to run with children, the reverse. I'll adjust my usage to metrics.

(I will only rate a game on metics, if it's set in ancient Athens.)