Exclusive details on the DDI API (not a repeat...)

[tomBitonti]

Originally Posted by Asmor

Stop getting caught up in legalese. Seriously.

As long as you're not doing it for profit, you won't get sued. I promise. Worst case scenario, they'll send you a C&D.

... snip ...

It is absolutely clear that this is what Wizards of the Coast wants people to do.

IMO, this is bad advice. Working for a very major corporation, and having mandatory yearly education on this sort of stuff, generally, when in doubt, the first step is to stop and consult the legal team.

In this case, I don't think Mr. Rouse's posting is sufficient to present the WotC policy. I would be careful until the online agreement and until the posted policy on the website are updated. Mr. Rouse's posting is a positive sign, but until there is an update that is clearly from the WotC legal team, some care should be used. (I'm actually surprised at the posting the Mr Rouse has made: That could put WotC in some danger, and is the sort of statement that usually has to be put through legal clearances. I can't tell if that has been done. If it has, my apologies. In this case, absent a very clear statement that the policy has legal approval, I can't accept it as a definitive statement. Also, this is an area where you really don't want multiple streams of statements. That muddies the water where you want very clear policies.)

Basically, I'm looking for an official statement that appears on the WotC website that can be provided through a link.

[PostScript: Under the DMCA, a Cease and Desist is not nearly the worst case scenario.]

Thx!

TomB

[Asmor]

Originally Posted by miscreationist

Yeah, I guess you are right. That is what has been done thus far is to send C&Ds. But I figured they'd go to account banning before that if you were automating the extraction of data and they didn't like it. The only reason I had not taken an interest thus far was to avoid getting my account removed for automating access to it - something I'd not like to happen if I'd paid for an extended amount of access to DDI.

Thanks for bringing up the topic. Hopefully, this is a good step and as I said, I look forward to the upcoming article about it.

It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.

I can send you to the page, but you still need to log yourself in.

Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.

[miscreationist]

Originally Posted by Asmor

It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.

I can send you to the page, but you still need to log yourself in.

Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.

Right. I understand that this is totally different from screen scraping and it is clear that they are documenting the API. I took a look at the Compendium Search service and noticed that everything accessible through its operations is data that is accessible to the public without having to log in. If there is no access to data that requires a login then I suspect the legal questions still stand about automated access to it (and I suspect that operations like that are against the TOS). I hoped that the next article might provide access (with authentication) to the data through the API.

Here's to hoping.

[tomBitonti]

Thinking more about the access, there are a couple of access types that seem to be problematic:

1) Anything that does queries and caches the results;
2) (1), where the cache is made available to other people.
3) Providing a forwarding service, where a third part contacts you through an API of your design, which prompts your API to log into the for-pay DDI using your login information to complete the request.

In a group environment, I can see (3) being very useful, where the DM and players each has network access, and where the DM has a portal to the DDI, and where the players access the DDI through the DM's portal. That seems useful enough, and simple enough to implement, that folks are very probably already thinking about doing it. That also seems to be exactly what WotC would not approve of.

Note that a web-site may have a policy that prohibits access to secondary pages without your having visited initial pages, and this is (?) actionable (?) enforceable, and has been tested through case law.

[malraux]

Originally Posted by Asmor

It's worth noting that I don't actually get access to any data behind the paywall... All I'm getting is the search results anyone can find, and the links anyone can find.

I can send you to the page, but you still need to log yourself in.

Now, theoretically it would be possible to log in and scrape the database (and, indeed, I've been in contact with people who have claimed to have done just that in months prior), but that doesn't require the API at all.

Yeah, I could code up an automator script to get safari to pull everything down, but that would still be my web browser getting the info.

On a related note, have you figured out what the code is to log in. obviously you'd want to post email and password but I still haven't figured out where that goes.

[Asmor]

Originally Posted by malraux

Yeah, I could code up an automator script to get safari to pull everything down, but that would still be my web browser getting the info.

On a related note, have you figured out what the code is to log in. obviously you'd want to post email and password but I still haven't figured out where that goes.

Looks to be login.aspx

<form name="form1" method="post" action="login.aspx?page=monster&amp;id=2351" id="form1">

Here're the relevant bits:

<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWBAK5u9HLAQKyzcaDDQLyveCRDwK4+vrXBTI0l5j0hL3ej51 4UF0fpXH07GN4" />
<input name="email" type="text" id="email" />
<input name="password" type="password" id="password" />

[miscreationist]

Asmor, from your article, http://www.wizards.com/dndinsider/co...rs&tab=Monster

Using + works fine.

[Asmor]

Huh. That's strange. I swear I tried + and it didn't work. o_0 Thanks.

[miscreationist]

The second part of the info on the API went up D&D Insider: Compendium API - Part 2 yesterday.

One thing to note is that in my testing, null is needed instead of NULL in the filters. I saw the KeywordSearchWithFilters method the other day, but now that the filter strings have been disclosed, it makes it really useful.

Apparently they will be disclosing how to get to the data in a third article which addresses "security issues and concerns". This does indeed look promising and it is clearly meant for automation, but I still think that their TOS conflicts with the use of this. Perhaps it will get a refresh.

[Asmor]

Originally Posted by miscreationist

Apparently they will be disclosing how to get to the data in a third article which addresses "security issues and concerns". This does indeed look promising and it is clearly meant for automation, but I still think that their TOS conflicts with the use of this. Perhaps it will get a refresh.

There's a fanpage policy in the works... Should have been out a while ago, frankly, though given the timeline of the vastly-more-important GSL I'm not all that surprised by the delay. My guess is that it's "in legal," since that seems to be the usual refrain.

[catsclaw227]

I am getting an application error when accessing the Webservice. Typical .NET error if the custom errors aren't set up.

Is it currently down or inaccessible, or is the webservice itself not working unless you are logged in?

I have logged into the DDI and then in a different browser tab, gone directly to the webservice and used the KeywordSearch method with keywords="Human Guard" and tab="Monster". Even trying an HTTP GET and I get the same error.

[Asmor]

You don't need to be logged in to perform searches.

Double check and make sure the service is actually working. I've found it has a knack for noticing when I want to work on a program and it decides to stop working for a while. :/

[catsclaw227]

As it turns out the WSDL the webservice has tied to it, brings back incorrect URLs for the service.

  <wsdl:service name="CompendiumSearch">
    <wsdl:port name="CompendiumSearchSoap" binding="tns:CompendiumSearchSoap">
      <soap:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchSoap12" binding="tns:CompendiumSearchSoap12">
      <soap12:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchHttpGet" binding="tns:CompendiumSearchHttpGet">
      <http:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
    <wsdl:port name="CompendiumSearchHttpPost" binding="tns:CompendiumSearchHttpPost">
      <http:address location="http://www.wizards.com/compendiumsearch.asmx" />
    </wsdl:port>
  </wsdl:service>

[Asmor]

Good call, catsclaw. I was able to get it working by specifying the address myself, e.g.

var c = new CompendiumSearch.CompendiumSearchSoapClient("CompendiumSearchSoap12", @"http://www.wizards.com/dndinsider/compendium/compendiumsearch.asmx");

[catsclaw227]

Yea, I ended up doing that too.

Also, playing around with the API without creating a webreference to the DDI service, I also used a simple approach and just loaded the url into an XDocument. Then I can easily traverse the monsterlist with Linq to Xml.

    XDocument monsterXml = GetSearchResults("http://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx/KeywordSearch?Keywords=Human+Guard&tab=Monster")


    public XDocument GetSearchResults(string url)
    {
        XDocument resultsXml = new XDocument();
        try
        {
            resultsXml = XDocument.Load(url);
        }
        catch (Exception e)
        {
            throw e;
        }
        return resultsXml;
    }

[catsclaw227]

I am using the API to be able to take the monster data and convert it into a fully functional rptok file for MapTools and the 4e framework I am using (Rumble's).

Still, it's a work in progress, and I am hesitant to write an HTML parser for the monster.aspx page because they are likely going to release the individual elements (like monster) API soon and I don't want to waste too much time writing a throw-away.

[lacan2002]

Hey, great thread! Any ideas on a general place to talk about DDI Api?
Seems like there are a number of interested D&D player/programmers out here, it would be nice to have a place for the hobbyist to share ideas.

It looks like they've added a needed NameOnly parameter, which I've just been setting to "false".

Thanks very much to Asmor's examples, I got a simple ruby client working.

For the interested, this requires the excellect xml-object and restclient gems, but once those are ready, getting an implicit object model that allows you to deal with the meat of the data is dead simple:

xml = RestClient.get('http://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx/KeywordSearch?Keywords=mind+flayers&tab=Monster&NameOnly=false')
data = XMLObject.new(xml) # That's it! Gives us a model implicitly!

# get each monster and show some data
data.Results.Monster.each do |monster|
  puts "#{monster.Name} is found in #{monster.SourceBook}"
  # do something more interesting, like put into a ul in HTML
end

The results for this look like this:

Mind Flayer Infiltrator is found in Monster Manual
Mind Flayer Mastermind is found in Monster Manual