ENnies V - and beyond...

[Conaill]

Wow, so many things to comment on. Let me just go through the thread and tackle them one by one:

To date the judges have been drawn from the ENWorld site. The membership is principly d20 so some may argue that the awards will be slanted to d20. To deflect this, I propose allowing other sites to nominate some of the judges.

Art & Cartography Categories: These two categories should have their own judges seperate from the other 8 proposed above.

Protestations from the current judges notwithstanding, given the number of judges and number of categories, it's unlikely that all of the judges would be equally knowledgeable about all the categories. I'm not assigning any blame here, mind you! I think the current and past judges have done a wonderful job. It's just a fact of life that most people tend to be most interested in a smaller set of categories, and we shouldn't necessarily require people to be up on *all* of them, as a prerequisite to be a judge for the Ennies.

Perhaps judges could indicate which categories they would like to judge. If you prefer well-rounded judges, you could require them to sign up for at least 5 categories (so you don't get an art judge who doesn't know squat about roleplaying, for example). It would make the judge selection process slightly more complicated, but as the Ennies' scope becomes larger, I think it's worth considering this.

(Note that this can be an entirely *separate* issue from which community the judges should be drawn from. I do think we have enough in-house expertise to judge most of the categories.)

The biggest need we have right now is to pare down the categories a bit. [...] Now, let's say you think a new division is essential. For each such proposed division, ask yourself which existing division you are going to remove, and how that is going to be a more functional division than the existing one.

Roleplaying Game
Setting
Adventure
Supplement
Product Line
Small Publisher
Large Publisher
Cartography
Interior Art
Cover Art
Graphic Design & Layout
Independent Project

Cartography, Interior Art, Cover Art, and Graphic Design & Layout? That's 4 out of the 11 awards devoted to "looks" over "content". I would greatly favor reducing the art&layout categories to two or three, and making room for one or two more "content" categories (d20/OGL vs non-d20/OGL for example, or fantasy vs sci-fi).

Dextra's later list still has 4 out of 16... a much better proportion, but it doesn't exactly follow Psion's recommendation to remove one category for every one you add. If we do shrink it down from 16, one of those 4 should probably be the first one to go.

Cartography and Cover Art are the two most obvious ones, because in most cases that boils down to ONE single piece of art. I'd much rather have an "artist of the year" or even an "artpiece of the year" award than an "artpiece of the year - cover art" and an "artpiece of the year - cartography" award...

I personally think there are better ways to get other communities involved without relinguishing control of the judges. Just getting other sites to post prominant links to the voting booth would help dramatically.

I think this is CRUCIAL, as long as the EN World is still very much d20/OGL dominated (and make no mistake about it, right now it *is*).

I missed the slide show. For those not in the know, SpoonyBard's laptop went foom, hence, no show. Next year, let's just bring a CD (or two) with the content so this doesn't happen again.

I'd love to have some sort of "processional" music for when recipients are coming to/from the stage. The dead air after each award was announced wasn't great for the atmosphere. Some pre-show and post-show music serving as a cue for when the show begins and ends, as well as getting poeple in a party mood, would be a valuable addition.

Totally agree on both these points. Last time I gave a presentation to more than 10 people, I made sure I had a CD backup and one on a USB memory stick (plus I made sure I had a downloadable copy online as well). Never trust your computer to behave for any sort of presentation or demo.

As for music, there's quite a few people putting out rpg related music or other audio. Doesn't matter if theyve been nominated for an Ennie or not. I'm sure if you contact the people putting out the RPG Sound Mixer or even Baldur's Gate, they'd be willing to provide background music in exchange for a little free publicity. And by all means, do put up that "GM Hold Music" during the intermission!

The current voting scheme favors a minority of partisans who have a specific loyalty or hatred. [...] A voting system should never benefit people who want to game it -- and the current one does.

I have to agree with Aaron here. Although it *seems* like the current voting procedure is more fair because it allows you to vote only on those products you are familiar with, it has some very serious drawbacks! It seems to me that this system was chosen on a somewhat ad-hoc basis, without much regard to current understanding of voting theory etc.

In particular, allowing people to vote for only one single choice opens up some very serious imbalances, as Aaron pointed out. Secondly, allowing people to assign a 1-10 "weight" to each vote also makes it a lot more "abuseable". For example, if you really want one specific system to win, the best voting strategy is to vote "10" for that product, and "1" for all the others in the category. It is always in your best interest to vote for *all* the products, even if you are not familiar with them!

Aaron mentioned some alternatives. At the very *least*, you should only count people that have voted for 2 or more choices in a category. Perhaps have people rank the products instead of assigning an unconstrained score (so if you only vote for 2 out of the 5 choices, ranking them 1 and 5 would have the same meaning as ranking them 1 and 2; and "ranking" only a single choice is meaningless). One-Person-One-Vote is ugly, and I'm not sure it would be better than the current scheme. Personally, I'd prefer approval voting (check off as many products as you want) over OPOV.

 

[AaronLoeb]

One-Person-One-Vote is ugly, and I'm not sure it would be better than the current scheme. Personally, I'd prefer approval voting (check off as many products as you want) over OPOV.

I agree. OPOV, as I said, is a lousy system -- but at least it favors the majority opinion; the current system favors the minority opinion. Approval voting would be much better, you are right. You vote for everything on the list you think deserves the award and every vote is equally weighted. The problem with it is that it still benefits the people who want to game the system, making the votes of people who vote for only one product count more because they aren't "dilluting" their vote. If you do approval voting where you have to vote for at least three products, that would solve that gameability problem, but at that point you might as well use ranked votes with points attached (see below).

My first post was very, very long. So, in case folks skipped it due to extreme verbosity, here are the suggestions in brief:

1) Close voting only to EN World members to limit outside "partisans" who are more likely to desire to manipulate the voting results ("I've come here only to vote for company X because I want company X to win").

2) Change the current voting system, which is emminently gameable and makes it not only possible, but likely, for people on the fringe with extreme opinions (positive or negative) to skew the results. Instead, go to ranked point-based voting system where you must rank at least 3 of the 5 products. Your number 1 choice gets 5 points, number 2 gets 4 points, etc. The reasons this would work better are argued in my previous inexcusably long post.

I believe these two changes would go a long way toward making the awards more unquestionably in line with the EN World community's opinion and would make it harder for outside influencers to radically alter the results.

 

[RangerWickett]

I'm afraid my internet will give out on my soon, so I'll make this brief.

First, I'd like to suggest we close this thread and open a new one, consolidating all the good ideas we've had, and clearing out those comments made that might come across poorly to people who haven't read the whole thread. Since a lot of people seem to be digging Denise's (Dextra's) ideas, it will behoove us to have them earlier in a thread, so people who just pop in can comment on them.

Second, I heavily encourage all publishers who are nominated next year to host 5 to 10 page pdf samples of their nominated products on their sites, or to put up galleries of art. The ENnies site would list the nominated products, provide a description of the product, its publisher, its product line, and maybe a two-sentence explanation of why the judges picked it.

Also listed would be links to samples of nominated products. That way, hopefully no voter would have to be entirely uninformed. I'm not sure if we'd want to include some sort of option like "Click here if you have only read the publisher's hosted sample" so that votes based on just that would be weighted less, or if instead we'd have everyone pick their primary choice (worth 3 votes) and their secondary choice (worth 1 vote).

Third, as a member of EN Publishing, I'm gladly willing to take on some of the responsibility of making the awards run smoothly. I'm not as much of a micromanaging ***** as Denise (*grin* I love you Denise *grin*), but I've got my fair share of nitpickiness. Also, heck, it might allow us to have one or two ENPub products at the ENnies booth, if no one objects.

So:

1) Move back the date to submit products to March 31st. Determine judges by January 31st. Require publishers to send products to judges directly. Additionally, the publishers must be able to deliver product images of anything they submit to the web boys at ENWorld. Entry fees seem like they'll be necessary, but they'll be modest.

2) Once the top five are selected, we send out an email and a snail mail letter to the point of contact for the companies whose products have been chosen. Those companies will be asked to RSVP, and will be heavily encouraged to host samples of their products on their sites in easy-to-find locations (and to send links to us).

3) ENnies voting should start probably in the middle of July. Do it after Origins and the US 4th of July holiday, but early enough that we'll be able to have everything set up and ready to go by the end of July. Actually, remind me what date GenCon 05's gonna be?

4) The ENnies booth should preferably not have a huge honking board shielding it from half the people walking past us. We should have a banner, a computer with internet access (wireless?) so people can see the site, and a slide show demo of some sort showing the nominees. We will have a list of the nominees, and a nicely bound folder with a page devoted to each award, so people can flip through it. I'm going to encourage EN Publishing to prepare 'sampler CDs' with copies of all of our pdf products, which we can maybe raffle off or something, to encourage people to come to the booth. Gamers love swag, and our booth lacked swaggage.

5) The ENnies ceremony. Yes to a preshow cocktail thingy. Yes to more formal attire and ushered seating. Yes to an after party.

Okay, so not that short of a post, but those are my comments and ideas.

Oh, wait, product categories.

I am fully in favor of opening the awards to more than just d20, because GenCon likes us and we share in their fame. However, we do share, so we should keep some of our own focus, and thus we cannot completely do away with one or two d20-only categories.

 

[Psion]

Some have said that not much voting theory went into our system and it favors the minority.

I find this strange, because, according to one rather interesting article on voting theory I ready, the system that we use in POLITICS, One Person One Vote with the possibility of block voting (~~sound familiar?~~) has a tendency to break up the Majority (the article I refer to shows how John McCain was the most popular candidate in 2000, but how our voting system shut him out.) At the same time, Aaron here says that our system favors the minority, yet I saw the votes rack up and did not find the votes exceptionally skewed. Those who got the vote out, won. The only effect our system had, as far as I could tell, is to help factor out unfamiliarity.

However, I do agree it could be simpler. We are essentially using a system that is best for people familiar with a topic, but does not require people to vote. Sound familiar?

If you ever refer to IMDB (a place as filled with extreme opinions as any. Go look there and see the slant to 10's and 1's.) it ought to.

However, they use some mathematical tricks we don't. And we ought to.

I'll see if I can dig up my stuff on voting theory, but in the meantime, I would ask Aaron why he thinks the minority view is catered to, because I am genuinely not clear. If 1000 people vote their favorite product 10 and one they don't like as a 1, then their view is going to prevail over 500 who have the opposite view. The only difference is that if you aren't familiar with a product, you aren't forced to weight down or bouy it's score with your vote.

If discontinuity with a statement of purpose is the problem, that can be tended to. I think the goals that went into the system are worthy, and I could try to dredge them up and distill them down.

We do agree (especially after talk of a supposed sneak peak at gaming report) that we need a better, more secure script.

 

[RangerWickett]

I'm afraid my internet will give out on my soon, so I'll make this brief.

First, I'd like to suggest we close this thread and open a new one, consolidating all the good ideas we've had, and clearing out those comments made that might come across poorly to people who haven't read the whole thread. Since a lot of people seem to be digging Denise's (Dextra's) ideas, it will behoove us to have them earlier in a thread, so people who just pop in can comment on them.

Second, I heavily encourage all publishers who are nominated next year to host 5 to 10 page pdf samples of their nominated products on their sites, or to put up galleries of art. The ENnies site would list the nominated products, provide a description of the product, its publisher, its product line, and maybe a two-sentence explanation of why the judges picked it.

Also listed would be links to samples of nominated products. That way, hopefully no voter would have to be entirely uninformed. I'm not sure if we'd want to include some sort of option like "Click here if you have only read the publisher's hosted sample" so that votes based on just that would be weighted less, or if instead we'd have everyone pick their primary choice (worth 3 votes) and their secondary choice (worth 1 vote).

Third, as a member of EN Publishing, I'm gladly willing to take on some of the responsibility of making the awards run smoothly. I'm not as much of a micromanaging ***** as Denise (*grin* I love you Denise *grin*), but I've got my fair share of nitpickiness. Also, heck, it might allow us to have one or two ENPub products at the ENnies booth, if no one objects.

So:

1) Move back the date to submit products to March 31st. Determine judges by January 31st. Require publishers to send products to judges directly. Additionally, the publishers must be able to deliver product images of anything they submit to the web boys at ENWorld. Entry fees seem like they'll be necessary, but they'll be modest.

2) Once the top five are selected, we send out an email and a snail mail letter to the point of contact for the companies whose products have been chosen. Those companies will be asked to RSVP, and will be heavily encouraged to host samples of their products on their sites in easy-to-find locations (and to send links to us).

3) ENnies voting should start probably in the middle of July. Do it after Origins and the US 4th of July holiday, but early enough that we'll be able to have everything set up and ready to go by the end of July. Actually, remind me what date GenCon 05's gonna be?

4) The ENnies booth should preferably not have a huge honking board shielding it from half the people walking past us. We should have a banner, a computer with internet access (wireless?) so people can see the site, and a slide show demo of some sort showing the nominees. We will have a list of the nominees, and a nicely bound folder with a page devoted to each award, so people can flip through it. I'm going to encourage EN Publishing to prepare 'sampler CDs' with copies of all of our pdf products, which we can maybe raffle off or something, to encourage people to come to the booth. Gamers love swag, and our booth lacked swaggage.

5) The ENnies ceremony. Yes to a preshow cocktail thingy. Yes to more formal attire and ushered seating. Yes to an after party.

Okay, so not that short of a post, but those are my comments and ideas.

Oh, wait, product categories.

I am fully in favor of opening the awards to more than just d20, because GenCon likes us and we share in their fame. However, we do share, so we should keep some of our own focus, and thus we cannot completely do away with one or two d20-only categories.

 

[Psion]

Another thing I would add:

I would dispute that "Cartography" is not content. In fact, I would be loath to release any of the graphical categories. I think they all play an important role in the value of a product.

 

[Umbran]

At the same time, Aaron here says that our system favors the minority, yet I saw the votes rack up and did not find the votes exceptionally skewed.

Well, let us not confuse past performance with future possibilities. The fact that the system has not been abused does not mean that it cannot or will not be abused. This becomes more important as we consider encouraging folks from elsewhere to come and vote.

Those who got the vote out, won. The only effect our system had, as far as I could tell, is to help factor out unfamiliarity.

Where I come from, "getting out the vote" is campaigning. If that's what you mean, well, I don't think that is what we want to see in our voting.

There are other systems that help factor out unfamiliarity without allowing anyone to skew averages.

 

[Psion]

Where I come from, "getting out the vote" is campaigning. If that's what you mean, well, I don't think that is what we want to see in our voting.

Getting the vote out is telling your fans to go vote, as far as I can tell.

And yes, that gives it the fundamental quality of a popularity contest. I hold that you can't factor that out and have it be a "people's choice" award. In fact, to try to do so defeats the purpose. It's like saying "we wanna know what you think, but we are going to ignore you."

There are other systems that help factor out unfamiliarity without allowing anyone to skew averages.

I would be interested in hearing about them, because you have me at a loss.

 

[AaronLoeb]

Some have said that not much voting theory went into our system and it favors the minority.

I did not mean to imply or say that not much voting theory went into the system. I assume thought went into it, since thought has obviously gone into every other part of the ENnies. I believe there are alternative systems that would better achieve the goals this system was put in place to achieve. Repeated: I think the ENnies have been very well planned and very well executed and I think you guys have been very smart about them at every step. So please do not read my comments as derission for your considerable and impressive work.

One Person One Vote with the possibility of block voting (~~sound familiar?~~) has a tendency to break up the Majority (the article I refer to shows how John McCain was the most popular candidate in 2000, but how our voting system shut him out.)

I agree wholeheartedly. OPOV is a lousy system. I only think it is less gameable than the current system -- a theory on my part that may be wrong.

yet I saw the votes rack up and did not find the votes exceptionally skewed. Those who got the vote out, won. The only effect our system had, as far as I could tell, is to help factor out unfamiliarity.

I can't speak to that as I have not seen the voting results. My comments are directed at a potential for abuse that may have never happened. Without looking at voting results, I can't argue this point about past results -- I'm addressing my comments to a potential problem. As the awards gain in popularity and stature, I think a new voting system would be a good idea.

However, I do agree it could be simpler. We are essentially using a system that is best for people familiar with a topic, but does not require people to vote. Sound familiar?

If you ever refer to IMDB (a place as filled with extreme opinions as any. Go look there and see the slant to 10's and 1's.) it ought to.

It is an excellent goal and a worthy example. The problems with the comparison are: 1) that the IMDB's rating system is meant to find a ranking from 10s of thousands of users on a single item not in comparison to others 2) it is not a contest, so there is less incentive to game the system short of just raw, fanatical prejudice (which the Internet is filled with), and 3) if they were using an "average" system on the rankings, on more obscure movies you would see stranger results. This is why IMDB uses a weighting method rather than a raw average -- to avoid the exact kind of "vote stuffing" I'm worried about. They don't reveal how they weight votes so as to keep people from gaming the system. But were they using just an average or mean, as I think the ENnies are (please correct me if I'm wrong), here would be a good example of the possible problem I'm referring to:

Forbidden Zone is an obscure, cult-classic movie. Most people I've tried showing it to find it unwatchable. Most people haven't heard of it. Those who have are generally crazy for it.

Here you can see its raw votes on IMDB:

http://www.imdb.com/title/tt0080752/ratings

Because so many people have given this cult movie (that the average consumer, in my experience, finds unwatchable) a 10, its "average" score would be a 8.7. The weighted score is a 6.5. Again, I don't know how that weighting works. But let's go with this example. Only 587 people voted because most people haven't heard of it. And the overwhelming majority of those people were huge fans of this cult hit, so it got a mean score of 8.7.

Now, let's look at Fellowship of the Ring. I don't really know what anyone else's measure of quality for a movie would be, but I think the majority of us who have seen Forbidden Zone and Fellowship would agree that Fellowship is a better movie. I know, everything here is subjective, so let's leave quality out of this. Let's just think of Forbidden Zone as a movie with a small, fanatical following and Fellowship as an incredibly huge movie everyone has heard of.

http://www.imdb.com/title/tt0120737/ratings

134,363 people voted for fellowship. Over half of those gave it a 10 out of 10, but 6,344 of them gave it a 1 out of 10. Some portion of the voters wanted to pull down the score. Because of this, Fellowship's mean score is 8.6; lower than Forbidden Zone.

In this example, we can see a tiny, fanatical minority of people earning a movie no one else has heard of a higher mean score than a movie everyone has heard of and some people wanted to pull down (the dark side of popularity). Were you to go for straight means, the voting scheme has benefitted Forbidden Zone for being obscure.

However, they use some mathematical tricks we don't. And we ought to.

It is clear that they do, and unclear what they are. I think a simpler system of assigning points to products would be superior. Their mathematical tricks are necessary because the 1 to 10 ranking essentially allows people to give a product not only out-of-the-average positive votes but also out-of-the-average negative votes. Any system that allows people to vote "negatively" is likely to prove unfair as it will benefit people who dislike a company on principle rather than having any actual opinion about the product.

I would ask Aaron why he thinks the minority view is catered to, because I am genuinely not clear. If 1000 people vote their favorite product 10 and one they don't like as a 1, then their view is going to prevail over 500 who have the opposite view. The only difference is that if you aren't familiar with a product, you aren't forced to weight down or bouy it's score with your vote

If discontinuity with a statement of purpose is the problem, that can be tended to. I think the goals that went into the system are worthy, and I could try to dredge them up and distill them down.

My assumption is that the average EN World member -- likely the overwhelming majority of members -- vote 7s to 9s on products. I am a prolific lurker on these boards and find the people here to be reasonable and fair minded. It is because of this that I think the voting scheme is flawed because it takes the power out of their hands. I assume you don't get a 1000 people voting 10s and 1s on the same products. I assume you get 1000 people voting fairly average scores and then a few hundred voting 10s for their favorites and 1s - 3s for the competitors, or 1s for the companies they hate regardless of the quality of the product. This small minority can unfairly tank a product (by voting 1s) that the overwhelming majority of voters find to be pretty darn good or can elevate a product that the overwhelming majority of users have never heard of (by giving it 10s that are not offset by very many "normal" votes).

To use the Forbidden Zone example, if you have a product that most of the EN World community has never heard of (again, ignoring quality arguments -- maybe it deserves to win; that's not the point), it stands a very good chance of winning if it has a fanatical following. Why? Because the fair-minded people decline to give it any score. Then the fanatical minority comes in and gives it 10s. It then has a higher average than the products that most of the people who voted have heard of because they were all giving it "fair-minded" scores of 7 to 9.

Similarly, if you have a magnificent product produced by a company that a vocal minority loathes (in the case of Fellowship, above, because they didn't have Tom Bombadil or something), it can be taken out of the running by that minority. The overwhelming majority can give the product 8s through 10s, but if a large enough mean-spirited minority gives it 1s, it will be pulled down.

Put these two trends together and you have a system that has the potential (I don't know if it ever has) to take the power out of the hands of the average EN World voter and put it into the hands of outliers.

All that said, I am certain that the goals in creating this system were worthy. Why? Because I've seen that the goals of every other part of the ENnies have been worthy. This is not meant as an indictment of anyone's motives. This is meant as a highly wonky discussion of voting schemes. The potential damage of this system, even if it were to be horribly abused, is of course limited. Experience shows that the nominees are all of the highest quality, so it's not like bad products are walking away with ENnies as is often the case with awards programs in other industries. However, I think the voting system could be altered to unambiguously put the power into the hands of the majority of voters.

All of this is based on assumptions: 1) about voting patterns and 2) that you are not using some kind of weighting scheme already. If I am wrong, please hit me with the mighty rhetorical mallet of smackdown.

 

[Psion]

We are using a weighting scheme. However, I am not the one that implemented it, so I can't tell you how it works.

I do beleive that IMDB publicized their method. I beleive it's called a bayesian mean and is discussed on one of their pages. It's sort of interesting. I'll see if I can summon it up in a spare moment.