[ENnies] Categories

[AaronLoeb]

I have some thoughts on this based on running the biggest "video game press" award in the video game industry for a number of years (www.e3awards.com). These thoughts do not necessarily all apply, and I'm not trying to make a broad appeal to authority -- I just want to share experiences that may be of benefit. This is long-winded and wonky, but is an analysis of voting systems based on years of award-giving experience, so I hope you'll read it if interested.

When you have a ballot that has products on it that many of the voters may not have heard of, the best system, we found, was weighted voting where each voter MUST select at least 3 products in a given category. The reason being that it is highly unfair to allow people to vote for only one product on the list. It allows partisans and the lazy to pick the award rather than the well-informed. Let me explain:

If everyone is allowed to vote for up to 3 products with equal weight (just a checkmark next to them), the well-informed voters who pick three end up diffusing the impact of their votes. It is almost always the case that of any 5 nominees there are about 3 which are "favored," and they'll get the preponderance of the distributed, informed vote. Let's use a possible example from last year. Let's say in the "best setting" category, the informed voters all picked (and I list these three only because I assume they're the best-selling nominees of the category from last year) Oriental Adventures, Rokugan Campaign Setting and Ghelspad, with some variation to the other two nominees that end up cancelling each other out. The result being that the "informed" segment of the voting populace managed to give these three products (totally random number) 5,000 votes each give or take. Now enter the partisans and the uninformed who, rather than vote for 3 prodcuts, will vote for only one. The partisans will vote for their favorite company more than their favorite product. S&S and AEG muster what partisans they can, but can anyone doubt that the force of wizards.com pointing people to a ballot simply trumps any effort the rest of the industry can muster? Next you get the uninformed. These are the people vote based on the one product they own. As I think everyone can attest to, the overwhelming odds here are in favor of WotC.

The solution, we found, has to be in stymying the efforts of the uninformed and the partisan. You require every voter to rank their top 3 choices in the category. The product they rank as #1 receives 3 points, the product they rank as number 2 receives 2 points, the product they rank as 3 receives 1 point. The product with the most points at the end of voting wins.

To the partisan, this says: you're not allowed to simply pick your favorite company and vote. You have to actually think about this category and determine your favorite three here. If you can't do that, you're too partisan to vote. To the uninformed it says: you need to actually experience the splendid variety of the d20 market to vote for this award.

The argument levied against this mechanism is quite simple: "what if there really is only one product I consider worth voting for. I own them all and I hate all of them but one." This is where you have to put faith in your nominating committee that they won't pick 4 stinkers and one winner (which, based on last year's nominees, they clearly won't). If it actually comes to pass, despite the best efforts of the judges to come up with a hot ballot, that you just can't see voting for others in the category, then you just don't get to vote in that category. Suck it up. Democracy is hard. The best result of this mechanism is that it often ends up rewarding products that the overwhelming majority of people didn't put as #1, but that everyone, everywhere agrees is in the top 3. This is good, in my opinion, as it shows what product the majority of voters feel is a top product rather than giving power to the passion of a distinct minority.

There you go. Very, very long-winded post concluded. Sorry if it's too much.

Aaron

 

[Monte At Home]

I agree 100% with Chris on these two points:

I'd like to see products that have to play by the same rules compete against each other. I'm in favor of keeping the PDF categories, because PDF products have constraints that print products do not. I'm in favor of not seeing WotC compete in most categories because they are free of constraints the rest of us have to bear.

I think that there should be a pdf category. I think that products that have gone to print or are going to print should be exempt from them (like, for example, Malhavoc Press products). If there are going to be awards for best fan site and whatnot, as in previous years (which I think is cool), a best pdf award seems natural.

I also think that including WotC in the mix invalidates the awards. As Chris said, they have the D&D logo AND don't have to abide by the d20 license. Including their products in the same categories as say, d20 products makes about as much sense as including GURPS products or Warhammer miniatures.

(Not to mention that comparing their sales to a d20 publisher's sales is like comparing Walmart to a Mom&Pop store, further invalidating an award based on consumer familiarity.)

Now maybe the thing to do is to have a "Best D&D Book" category and let WotC and Kenzer slug it out there. Then they aren't totally excluded, but they don't get to sweep the awards again either.

Seems like a fine idea to me.

I absolutely LOVE the idea of a set of awards for d20 publishers only. It helps create a feeling of community among us all. I think including D&D products dillutes that.

Originally posted by Hellhound

With only 5 nominees, EVERYONE who doesn't win but who got nominated effectively got the silver medal.

While Nicole was swaying me with her argument, that is also a very good point. Hmm.

 

[FDP Mike]

d20 Pramas posted:
I'd like to see products that have to play by the same rules compete against each other. [. . .] I'm in favor of not seeing WotC compete in most categories because they are free of constraints the rest of us have to bear.

Monte At Home posted:
I also think that including WotC in the mix invalidates the awards. As Chris said, they have the D&D logo AND don't have to abide by the d20 license. Including their products in the same categories as say, d20 products makes about as much sense as including GURPS products or Warhammer miniatures.

And to think I deleted this very sentiment from my original post . . . .

This issue came up last year right after the ENnies, particularly owing to WotC's "sweep." I (along with others such as Clark Peterson) commented then that WotC is not "technically" a "d20" publisher, one reason being that WotC does not need to play by the same rule as the rest of us: the contribution of OGC in every product. Other than the SRD, the only "open" content WotC has provided appears, I believe, in Monster Manual II. Granted, WotC is the very reason that we're all here thanks to the SRD . . . but that's not the same. (I'd like to think of a good reason why right now, but I'm a sicky sicky . . . . )

Then again, not including WotC raises other issues for Morrus, such as perhaps the ENnies' place at GenCon. Still, I don't see why a smaller ceremony somewhere during GenCon could not be just as viable.

I really do see the arguments for both sides (WotC in or out), but my heart, my gut feeling has always told me that WotC, well, sort of shouldn't be in the ENnies until it makes a "d20" product in the sense required and intended by the OGL and d20STL.

If WotC, however, remains eligible, then I like this idea:

AaronLoeb posted:
When you have a ballot that has products on it that many of the voters may not have heard of, the best system, we found, was weighted voting where each voter MUST select at least 3 products in a given category. The reason being that it is highly unfair to allow people to vote for only one product on the list. It allows partisans and the lazy to pick the award rather than the well-informed.

Aaron's suggestion seems to me elegant, simple, and practical. I would really like to see this sort of system instituted for the voting, specifically because it solves many of the issues discussed so far about voting, as Aaron argues quite convincingly.

This system might also provide a "clearer" picture of the Top 3, if the ENnies go this route. On this note, I also like Morrus' idea of "counting down" the nominees and the percentage of the overall vote they received.

Morrus, is there any chance of instituting a voting system such as outlined by Aaron? It just seems to make so much sense . . . .

As for silver and bronze medals, my heart, my gut feeling tells me that I'm still not convinced -- especially if throughout the year publishers make a bigger deal out of receiving nominations, such as on their web sites and in their advertising (FDP included, by the way).

'Nuff said for now.


Take care,
Mike

P.S. I like Aaron's idea even if WotC is not made eligible for the ENnies. Just thought I should make that clear . . . .

 

[The Sigil]

Originally posted by Pramas
I'd like to see products that have to play by the same rules compete against each other. I'm in favor of keeping the PDF categories, because PDF products have constraints that print products do not. I'm in favor of not seeing WotC compete in most categories because they are free of constraints the rest of us have to bear.

Originally posted by Monte at Home
I also think that including WotC in the mix invalidates the awards. As Chris said, they have the D&D logo AND don't have to abide by the d20 license.

I agree here. This (different rules in development, not different rules in market penetration), IMO, is the only valid argument against including WotC. Market share will come to other publishers with time (and I think SSS and FFG are already making inroads), so the problem with market share will be cured as others inevitably rise to WotC's level (and, to be honest, I think WotC has helped by dropping off several pegs).

The thing that, in my mind, makes the playing field "un-level" is not market share, but rather that WotC, as has been so eloquently stated, gets to play by different (read: less/no) rules in developing their products and contributing useful material (read: OGC). To a lesser degree, this applies to Kenzer & Co as well.

I think that there should be a pdf category. I think that products that have gone to print or are going to print should be exempt from them (like, for example, Malhavoc Press products). If there are going to be awards for best fan site and whatnot, as in previous years (which I think is cool), a best pdf award seems natural.

To be honest, I'm not sure on this. My argument above stands - PDFs should not be given special consideration IMO based solely on "market share" or "underexposure." But on the other hand, PDF is a different medium from print - even if it "looks like" print in many ways. There are things you can do with a PDF that you simply can't do with a print product (not that I've seen many PDFs that do, but you could, for example, in theory create a PDF character sheet that "does all the math" for your character computing BAB and such - how do you put this in competition with a printed character sheet that simply can't do the same things?). PDFs are in their infancy, and they can do all sorts of things that a print product can't manage. I think with time they may begin to realize their full potential and there may indeed be a day when a PDF product is vastly superior to the print version due to the ability to customize it and make it flexible. That day is clearly not here yet.

While I'm not 100% sure, I think I have to lean toward being in favor a separate PDF category though - simply on grounds of being a vastly different medium.

Other notes:

1.) I think the "vote for 3 of 5" system proposed is a good one as well... it may help mitigate some of the market share problems discussed - and if Aaron is to be believed, it is a system that was already tried and proven to work for this purpose. In the spirit of OGC, let's "re-use" that!

2.) I think the idea of a publisher's choice award is a good one as well. Publishers have no real forum to pat each other on the back and it would be nice to have one.

3.) If this is to truly be a popular award, there is no need to exclude anyone from voting... that runs against the mentality of "popular" awards - but possibly at the expense of rewarding the "best" material.

Given #2 and #3 above, perhaps one solution might be to make the ENnies three-tiered.

Have one "popular" award open to voting by anyone. This includes ENWorld members and publishers, since they are subsets of "anyone."

Have one "active ENWorld/experts" (perhaps a misnomer) award open to voting on by anyone who meets certain critieria (possible criteria include such things as: post count, length of time since registering on ENWorld, anyone who has written 5 or more reviews on ENWorld or on other sites, etc. - whether you must meet one or many criteria is a matter for discussion). This includes publishers as well, since presumably many of them are a subset of "active ENworlders."

Have one "publishers" award where voting is restricted to publishers.

Thus, each category might have three awards...

"People's Choice" (sports analogy: NBA All-Star Team Starters, voted on by fans)
"Experts' Choice" (sports analogy: NBA All-Star Reserves and All-League Teams, voted on by coaches and sportswriters, respectively)
"Publishers' Choice" (sports analogy: NFL All-Pro Team, voted on by players)

Obviously, it would become a true mark of excellence to sweep these choices, but I think such a system might serve the purposes of multiple awards shows by incorporating the opinions of all three groups to whom attention is usually paid by awards shows. Each group's "seal of approval" carries a different sort of weight, but I think all three are valid things to strive for... I don't want to just be popular with fans, but want publishers to think highly of my stuff as well - and vice versa. As a buyer, for example, I might be more interested in Experts' Choice than People's Choice for instance - and I have no way of measuring that.

4.) Oh yeah, if RPGNow should work up a PDF awards system similar to the Ennies (and I think they should - here's me volunteering you guys to do even more work LOL), I would still be in favor of having one PDF award in the ENnies - unless RPGNow's awards become a subset of the ENnies.

EDIT: 5.) BTW, I don't count the OGC published in the MM2 - WotC was simply reprinting stuff from the Creature Catalog, not contributing anything new. 2 creatures. A big deal is made out of this, and unnecessarily.

EDIT: 6.) I would also like to see an award for "best OGC" or somesuch. Show us who is doing the most to "grow the pie."

Just my multiple cents.

--The Sigil

 

[Psion]

I like that WOTC is included. Sets the bar higher.

I don't like a Judges' Choice award. The nominations themselves are the Judges' Choice awards, right?

After a fashion, and this is something we sort of tried to impress on people last year, but it didn't really sink in and some publishers still express a sense of being slighted by being forced to compete with WotC despite being recognized by a nomination.

 

[Darkness]

AaronLoeb:

That's a very interesting system, IMO.

 

[MThibault]

"As for silver and bronze medals, my heart, my gut feeling tells me that I'm still not convinced -- especially if throughout the year publishers make a bigger deal out of receiving nominations, such as on their web sites and in their advertising (FDP included, by the way)."

If the value of a nomination is eroded then publishers will have to really promote their nominations *before* the awards to try to build/coalesce support to actually win one of the top three. After the awards are handed out, then really only the three winners will benefit from their association with the awards from that point forward.

I believe Hellhound when he says that he's just as proud of his nomination as his award. But the fact of the matter is that the award lets him put "ENnie Award winner" in his title or signature, on his website and even on the products. If being nominated is really as good as winning, then why announce a winner? Graciousness is a virtue and all, but you *want* the award itself to be more meaningful than the nomination.

Just two more cents I found next to the cold medicine (this bug really travels, apparently).

Cheers

 

[Eridanis]

I understand what Pramas, Nikchick, FDP Mike, and Orcus (second hand) are saying about WotC's involvement, but the fact is, they built the court we're all playing on. I think that gives them the right to an automatic seed in the tournament, as it were. (And I'm not convinced that they will sweep all the awards again this year, especially with one public ceremony, well publisized, under our belts, and people having better sense of what the awards are aiming for. There are too many good products out there for that, IMO.)

I don't like a Judge's Choice award (Wulf's exact comment sprung to mind as soon as I read about it), but if there is to be one, the best form I think it might take would be a kind of "Honorary Mention" Award. Here's how it could work. Morrus will know the results of the voting long before the awards ceremony. Once the voting is concluded, Morrus can go back to the judges, say, "Here are the winners of each category. Now take the four products in each catagory that didn't win, put them in a pile, and decide which of these products is the best, in your opinion." That would be the Judge's Award winner - recognizing a product that didn't win any awards, but was still good enough to be recognized as being apart from the pack.

I also like the "count 'em down" compromise Morrus suggested. If a publisher wants to put "came in second place in the 2003 ENnie award for Best Module", then they can, or they can just say, "2003 ENnie Nominee for Best Module." Or whatever. A medals system for only five nominees is overkill.

OGL award: good idea.

Writer's Award: Interesting idea, but now, how do you choose which freelancers will be the committee? A system that doesn't feel liek a popularity contest would have to be devised.

Of course, if things don't work out the ENworld's satisfaction this year (i.e. a WotC sweep that's perceived as undeserved), they can always be modified by implementing a voting system like Aaron described. It's a growing process, and each year brings new insight.

 

[Dextra]

couplov thoughts:

If somehow the market penetration/fairness issue manages to be resolved, then I'm not so keen about a PDF only category. I'm still torn on that one though- the nomination is what makes me happy, getting to compete with the big boys is a rush. But not so much of a rush as winning.

I love the idea of a best D&D product category.

If one of the factors affecting decisions about #s of awards handed out, I'd suggest doing medals, and hand out only a handful of trophies: one for People's Choice d20 over pagecount X, People's Choice d20 under pagecount X, People's Choice D&D, Judge's Choice d20 over pagecount X, Judge's Choice d20 under pagecount X, Judge's Choice D&D, special ENnie for Improvement of Open Gaming (where applicable). Hrm, maybe the publisher who wins the most medals could be tallied, sortov like the Olympics and could earn themselves a Publisher of the Year award trophy.

I really like the idea of medals- it's a way to distinguish ourselves from other awards, and recognize quality, not just popular vote.

Anyhoo, those are my current thoughts, just thought I'd muddy the waters a bit more.

 

[Blacksad]

AaronLoeb:

That's a very interesting system, IMO.

I'll second that, but I still prefer to give a note on each product.

On excluding WotC because they do not follow the same rules:
I DON'T CARE, I'M NOT A PUBLISHER!
The legal problem of publisher aren't my preoccupation.

The important part for me is: is this product worth of my € as a d20 player? and that's the reason why WotC should be included, plus this year, what they produced isn't near as good as what was made last year (because last year, it was really good), so I think that it's unlikely that we will see WotC winning as much prizes this year.