ENnies V - and beyond...

[PatrickLawinger]

Ennie Stuff

Okay, I'll admit it, the thread is 5 pages long and I have only scanned it. The site is slow for me, I am lazy, and frankly don't have much time. I have seen few publisher responses and I'd like to put forward another.

Funding: Okay, funding can't come from the companies/people submitting products. Sorry, it just can't. Even if you make it "voluntary" people won't feel that it is voluntary and any large donor that wins is going to spark discussion of corruption. Even if such charges are completely false, the taint on an award is something that should be avoided at all costs. To have a truly respected award the even the APPEARANCE of impropriety must be avoided. If you want to ask for fan donations, fine. If you are going to be the GenCon awards, then by all means make Peter pay for all of it.

Author/Artist/Cartographer Awards: you are asking for an entry fee. You are also asking them to submit pieces of work that may or may not be fully purchased by the publisher. Don’t do it. A well-written book has numerous steps involved, including editing. The “final draft” turned into a publisher by the author is generally owned by the publisher in this industry. Even if a writer wanted to submit, and pay the fee, legally most can not.

Award Categories: eh, whatever you want. Making the categories broader but doing this through a d20/DnD/OGL based website is simply going to marginalize the awards. Many d20 publishers won’t bother due to the breadth of the competition, and other publishers with other game systems won’t bother because they view this as a DnD site/award. Do one thing, do it well.

Shipping to Judges: I don’t know what others are going to say, but I am saying, um, no. Shipping costs can be very high, I admit that, and boxing individual sets of books, etc. takes time. It takes time from the publisher too. Most of the publishers submitting are going to be small, asking a small print publisher to ship 5-6 individual shipments instead of 1 per product is asking for a significant increase in the amount they need to spend to take part in the contest. Overseas and Canadian publishers might even choose to not to enter. If you want as many publishers as possible entered, make it as easy as possible for them to submit their products, don’t add additional steps.

The award itself: I am about to say things here that are sure to get me flamed. So be it. Some of the EnWorld members I saw at GenCon admitted to being surprised at just how huge the convention was, and how few people had even heard of EnWorld. I know people hate to hear it on these message boards, but in the overall rpg market, ENWorld is hardly a speck on the windshield. While a .pdf publisher with a following here might receive extra sales from a nomination or an award, print publishers hardly see a blip in sales. While most gamers use the internet, especially in this day and age, only a small percentage of those use it for gaming related questions, reviews, discussions, or even purchases. Avoid marginalizing the award by keeping it focused on DnD and related products, the further you expand the more you will be ignored, either because you are a DnD site voting on things you "don't know" or as an award that is simply overly broad.

Improving the Awards: the biggest improvement I see as being required is getting more publishers to take part, particularly WotC. Remember, big companies essentially have nothing to gain by winning a nomination or an award, but can certainly lose face if they miss out. Sure, nobody wants to compete with the “big dog” but if the “big dog” isn’t there what does the award really mean? Maximize the number of companies entered in whatever way you can. If the big guys don't participate, the awards mean very little. I think you should also work on timing, find out when the big parties are and make sure the awards don’t conflict with them. The White Wolf party was the same night and time as the Ennies award ceremony.

Respectfully,
Patrick Lawinger

 

[Umbran]

As for requiring a submission fee on top of sending product, simply put, that won't fly. Why would a publisher send 6 or more copies of each product for consideration (at a cost of at least $10 per book to the publisher) and turn around and pay more for the privilege of being considered for an award?

Well, why do they submit the product without the fee? Because they want recognition and/or the potential sales boost. Perhaps someone should go ask the publishers who submitted this year if they would continue to submit if there was a fee, or how large a fee they'd be willing to pay. This is a question that can be answered by sending out a few e-mails, so there's no need to sit around speculating.

Lastly, an unpopular point that has been brought up in the past but still remains an issue to publishers is the presence of a panel of judges to determine the nominees. Certain companies feel they will never get a fair consideration as long as judges with clear biases against those companies are involved in the selections.

In theory, perhaps. In practice, how many companies have done so?

The problem is that popular-vote nomination has an even worse problem - it is strongly biased against small publishers. I cannot nominate a product I have not seen. The panel of judges is guaranteed to have at least seen all of the entries, and has at least a chance to treat small presses fairly. Popular-vote does not.

One annoyingly inconvenient partial solution to the problems you mentioned is to not reveal who the judges are before the submissions are due.

Therefore, they will not enter the awards. This goes along with some of the criticisms that Monte Cook brought to light on his site's forums. Why should a publisher feel obligated to enter products for the ENnies, when he is publicly criticized on a regular basis by the EN community or on EN reviews?

Well, that problem goes beyond the panel of judges, and goes to the popular vote on the nominees, too. And that problem cannot be removed. No matter who is voting, a publisher may believe that the voting group is biased against them.

I haven't read Mr. Cook's remarks (do you have a link?), but my first guess is that the problem isn't a strong one. EN World picks it's judges (hopefully), based upon them being fair and reasonably capable of judging the quality of a work. If the judge was percieved as having a strong bias against one publisher, he'd be less likely to be chosen as a judge.

And, there's the fact that negative statements tend to get strongly printed on the human mind. We remember people saying bad things more than we remember them saying good things. How many individuals are actually responsible for such criticism?

 

[Conaill]

<Another post with nothing but voting systems wonkiness. Feel free to skip if this doesn't interest you...>

1) Regarding IMDB's voting scheme. They do *not* currently reveal the exact formula they use to calculate the score. Here's some comments from their page on this issue:

IMDb and IMDbPro display weighted vote averages rather than raw data averages. Various filters are applied to the raw data in order to eliminate and reduce attempts at "vote stuffing" by individuals more interested in changing the current rating of a movie than giving their true opinion of it. [...] In order to avoid leaving the scheme open to abuse, we do not disclose the exact methods used. [...] The scheme combines a number of well-known and proven statistical methods, including a trimmed mean to reduce extreme influences and, most importantly a complex voter weighting system to make sure that the final rating is representative of the general voting population and not subject to over influence from individuals who are not regular participants in the poll. The scheme has been developed internally over the 10 years which the poll has been in operation and tuned on a regular basis to make sure it remains fair.

After some digging around, I found out that IMBD originally did mention they were using a Bayesian estimate. They seem to have removed the formula from most of their web pages since, but you can still find it at the bottom of their Top Rated "Animation" Titles page:

The formula for calculating the Top Rated 50 Titles gives a true Bayesian estimate:

weighted rank (WR) = (v / (v+m)) x R + (m / (v+m)) x C

where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 50 (currently ??)
C = the mean vote across the whole report (currently 6.6)

It's important to point out that a Bayesian estimate doesn't help you get a more "fair" result in any way, it just helps you deal with the uncertainty caused by having a small number of votes (essentially, it adds a number of votes with average scores to get a more robust estimate of the population average).

The Bayesian estimation is a pretty simple trick which should probably be used anyway, regardless of what other manipulations are done to the score (assuming, in our case, that we do have some choices with a small number of votes, say less than 100). I assume imdb has since built in a lot more additional tricks to avoid vote stuffing. From their explanation, they probably throw away a few of the top and bottom % votes, plus it sounds like they may have a sophisticated way to estimate the likelihood that someone is manipulating the vote, downweighting those votes accordingly.

Do keep in mind that IMDB's goal with all of this is to get a score *number* for each "candidate" (i.e., movie), whereas the Ennies' goal is to pick a *winner*. Getting an actual score is a much harder issue, and we should be able to avoid an awful lot of the problems IMDB is forced to deal with. In particular, I strongly believe we should NOT let the voters assign a score, because that is where most of the abusability comes into play. Only allowing the voters to check of one (as in One-person-one-vote) or several choices (as in approval voting), or ranking the choices (as in instant runoff or other systems) gets rid of most of the power malicious voters have to manipulate the system.


2) Approval voting:

Approval voting would be much better, you are right. You vote for everything on the list you think deserves the award and every vote is equally weighted. The problem with it is that it still benefits the people who want to game the system, making the votes of people who vote for only one product count more because they aren't "dilluting" their vote.

Actually, people who vote for only one product have exactly the same amount of power as people who vote for all but one product, so "gaming the system" is not nearly as simple as casting more or less votes. In fact, the optimal "gamed" vote using approval voting relies on having an accurate estimate of the number of votes each candidate will get, and picking an optimal cutoff for the number of your favorites you vote for.

In general, the best strategy in an Approval system is to vote for the candidate you'd vote for in a Plurality system, and also to vote for all the candidates you actually prefer to that candidate. This strategy does not require you to rearrange the order of your true preferences; it only requires you to draw the line between your approved and disapproved candidates at a certain point in your personal preference ranking list.

3) Instant Runoff Voting

It seems to me that the Hugo Award voting procedure fits the bill.
------------
In summary -
You have some candidates in a cateegory. Let us say there are 5.

You may rank them in order of priority. You give priority 1 to the product you think is most deserving, priority 2 to the product next most deserving, and so on. I you are unfamiliar with a product, you don't have to vote for it at all. Just rank the ones you know.

You tally the votes for first preferences. If one product gets more than 50% of the votes, it wins.

If no product gets more than 50%, you go to the product that had the least first preference votes. You sort these by second preference, and those second preference votes are added to the previous first preference totals. If the ballot doesn't list a second preference, it is discarded.

Now, if one of the products has more than 50% of the remaining ballots, it wins.

Otherwise, we repeat the process, eliminating the least popular, and redistributing votes. If a ballot's 1st and 2nd preference have both been eliminated, you go to the 3rd preference, and so on.

This method is called Instant Runoff Voting (or IRV), because it's essentially equivalent to a bunch of runoff elections, each time throwing out the candidate with the lowest number of votes until someone gets a majority. It's coincidentally also the method the US Green Party has been proposing for years for local and state elections, because it tends to avoid the "spoiler" scenario. (Noet however that unlike Approval Voting, IRV still has a problem with "spoilers" as soon as there are more than two viable candidate).

You may have noticed I recommended ranking choices in my first post. IRV is one such system which uses ranked votes, but there are others, such as Aaron's suggestion of assigning point values to ranks (formally called "Borda Count"). Personally, I greatly prefer IRV over any point-based system, but I was a little worried about scaring off people with its complexity . Note that Borda Count is still subject to strategic voting as well, although it's less of a problem than when assigning poiunts freely. IRV also deals with missing votes (i.e. unfamiliar products in the Ennies) in a more natural way than Borda Count. Another ranked method, which is supposed to be even more fair than both IRV and Borda is the Condorcet method. Unfortunately, it's also a little more complicated...

In theory, how much a system can be exploited is directly related to how many different choices you leave to the voter. So in principle, a ranking scheme should be more "gameable" than one where you only pick candidates. For all of these reasons, and because of its simplicity, I generally prefer Approval Voting over IRV and other ranked methods. If it's good enough for the the Mathematical Association of America, and the American Statistical Association, it's good enough for me!

For some more info on Approval Voting, you can check here, here, or here These pages will also lead you to links to various other pages on voting systems...

 

[BastionPress_Creech]

Well, why do they submit the product without the fee? Because they want recognition and/or the potential sales boost. Perhaps someone should go ask the publishers who submitted this year if they would continue to submit if there was a fee, or how large a fee they'd be willing to pay. This is a question that can be answered by sending out a few e-mails, so there's no need to sit around speculating.

I can answer for one of those publishers right now in my official capacity as Marketing Director. Bastion Press will not enter any awards where we are required to "pay" for the privilege of being considered as a nominee. Sending six copies of each of the three titles we submitted represented a considerable dollar investment on our part. To tell us that we have to spend an additional $10-$20 per book to help subsidize the costs of the awards show is ludricous. If that is the course that the ENnies take, then we will no longer participate.

Money to run the awards cannot come from the companies that enter in terms of an admission fee. You want financial resources and backing to help with the awards? Make it through advertising and donations like other awards do. There has to be another way to finance matters other than an entry fee. This line of thought will only lead to less and less publisher participation until the awards degenerate into nothing more than a shadow of what they were.

 

[Psion]

Hmmm...

It seems like the idea of an entry fee is none too popular.

Would the idea of an "ennies fund" parallel to the normal ENWorld fund drive be objectionable to the natives?

 

[AaronLoeb]

<Another post with nothing but voting systems wonkiness. Feel free to skip if this doesn't interest you...>

NERD POWER!!!!

Actually, people who vote for only one product have exactly the same amount of power as people who vote for all but one product, so "gaming the system" is not nearly as simple as casting more or less votes. In fact, the optimal "gamed" vote using approval voting relies on having an accurate estimate of the number of votes each candidate will get, and picking an optimal cutoff for the number of your favorites you vote for.

Oh, cool! I had thought approval voting was simply that you give one point to every candidate you like and you add up the points -- highest point-getter wins. I will read the link you've provided! Thank you!

Edit: Okay, read through it... It looks like it's a sytem that's good for 3rd-party candidates, but my concern was valid. You add up everyone's votes equally. One may think that three products on the nominee list are good, but really wants the one his friend wrote to win -- so it's to his advantage to only vote for his friend's product and not vote for the others. I.e. it encourages cynical voting. Amplify this by "partisan" voting, where people get out the vote from consumers loyal only to them, and you'll have hundreds of ballots with only one product voted for and that one vote not offset by fair-minded votes for other products.

With this method, suppose there were three obviously great products on the list and every "normal" EN World voter gave each one approval: a vote. Now each one has about 1000 votes. Now the election will be decided by how many "outliers" there are -- people who vote for only one of the three. However, every system has this problem in one way or the other: the "get out the vote" effort can win the award.

The system encourages people to vote for obscure products they think won't win (as it does third-party candidates), meaning it helps with the unfamiliarity problem, but that doesn't mean it isn't gameable. It is still superior to the current system, I think, and is incredibly simple. It also gets rid of any possibility for a "negative" vote, which I think is a very good thing.

For some more info on Approval Voting, you can check here, here, or here These pages will also lead you to links to various other pages on voting systems...

This was an awesome, thoroughly informative post. You are my new king.

 

[Monte At Home]

For what it's worth, assuming that the changes made (or not made) to the ENnies resulting from this discussion present an awards that we want to be a part of, Malhavoc Press would be willing to pay a small entry fee in order to help defer what I assume must be considerable costs.

 

[Fiery James]

I'm sure that if we all decided to crash at Fiery Dragons booth, they wouldn't be too happy with us, as they wouldn't be able to promote their products.

We'd just charge admission.

And, of course, you're always welcome to share space with Hal.

- James

 

[Umbran]

I can answer for one of those publishers right now in my official capacity as Marketing Director. Bastion Press will not enter any awards where we are required to "pay" for the privilege of being considered as a nominee.

Fair enough.

Money to run the awards cannot come from the companies that enter in terms of an admission fee.

I'd suggest you try to refrain from telling us what we can and cannot do. Try to stick to "should and should not" or "that would be unwise". Trying to dictate policy to volunteers isn't a winning strategy. I'm sure advice and opinions are welcome. But demands usually aren't.

You want financial resources and backing to help with the awards? Make it through advertising and donations like other awards do.

Fine, would Bastion Press like to make a donation?

The thing is that the gaming community is pretty small. It has already been seen that ads and donations are not always enough to keep the site afloat, much less also cover the costs of the awards.

For what it's worth, assuming that the changes made (or not made) to the ENnies resulting from this discussion present an awards that we want to be a part of....

Which brings up another question - do we want to poll publishers to see what they do and do not object to in the awards? While ultimately the Ennies must be something we fans like and approve of, there's something to be said for making the awards publisher-firendly as well.

 

[DaveMage]

Would the idea of an "ennies fund" parallel to the normal ENWorld fund drive be objectionable to the natives?

Speaking as a fan and not a publisher, I have no objection to a fund drive.

But there's no way in hell I'm donating money to an award show which doesn't benefit me (i.e. Joe Board Member) in the slightest. [Note, the *show* does not benefit me, although the awards themselves might if they help me with purchasing decisions for d20 products.]

Now, if the ENnies come up with a mission statement that shows me how these awards might benefit me, I may change my mind.