Open Gaming Content databases

[Morrus]

Here's a preview of the Kickstarter for it all; it's not finished; things will change; all the numbers are still being worked out. But it's a start, and I'd love some feedback. The hard part right now is the darn rewards; the idea that the product be free makes it hard to offer it as a reward, and I'd like something better than the traditional vanity-style 'get your name in the credits' type thing. Anyhow, please take a look and offer your thoughts. It's only a teeny, tiny little Kickstarter - it'll only need a few hundred, so I'm not attempting any records here!

http://www.kickstarter.com/projects/enworld/41999025?token=26db7ab9

 

[Janx]

Here's a preview of the Kickstarter for it all; it's not finished; things will change; all the numbers are still being worked out. But it's a start, and I'd love some feedback. The hard part right now is the darn rewards; the idea that the product be free makes it hard to offer it as a reward, and I'd like something better than the traditional vanity-style 'get your name in the credits' type thing. Anyhow, please take a look and offer your thoughts. It's only a teeny, tiny little Kickstarter - it'll only need a few hundred, so I'm not attempting any records here!

http://www.kickstarter.com/projects/enworld/41999025?token=26db7ab9

Looks cool.

As I'm a programmer (not involved with this project, but interested in it), here's what I think is being described:

From what I gather, when I visit the new Databases page, I'll pick a game system and database. Presumably the Database choices might be Items, Spells, Feats, Skills, Monsters.

Once I pick a Database (might call the Categories for the mundanes), I expect I can peruse the entire list in a GridView, or search some of the main fields of that category (name, HD, spell level, etc, free-form text?).

Note, I hyperlinked GridView to a product I use for such (once folks see it they'll know they've seen these before but didn't know what us developers call them). That's not a sales pitch, EN World will be using something else, I just wanted to give an example.

Once I look at this GridView (aka fancy table) of results (say 2HD monsters in the Temperate climate), I can double-click (or something) to view the details of the selected item in a pop-up (or new screen)

Because each Category/Database is different (spells need different fields than monsters), I expect to see different columns in the GridView and in the Detail View, per the respective category chose and item selected. Within that category (ex Monsters), they will of course all have the same fields.

To also clarify, it is assumed that Game System is an aspect of Category, in that Monsters doesn't really differentiate between PathFinder monsters and ShadowRun monster stats. So I expect these databases are qualified by a category name and a rule-set, if nothing else before getting into the goodies.

That would seem to define what users will probably see, based on my understanding of the project (I read what you read, and I know what I know about building stuff like it).


the interesting part of the project involves category/database definition. A user has the ability to create a new database on their own (ex. I want to create a Monsters database for AD&D2e using content from the old Tools CD that TSR released in RTF format).

I will have the ability to add a new rule set (AD&D 2E), and then add a new Category for "Monsters". Then I will be able to define the fields needed (the stat block sections from the Monsterous Compendium). I will get to choose what order these fields show up on the screen, and possibly their datatypes (short text, numbers, big box of text with formating).

I know from experience, this is where the tricky part comes in. There's a few different ways to program this, one of which is to let the code dynamically create tables in the database. Which is potentially dangerous (i.e. security risk). On the other hand, it will result in more performant code (runs faster, loads faster). I'd be curious as to how EN's developer is tackling the problem, as I've done quite a bit of work in dynamic data systems in database applications.

 

[Morrus]

Kinda. Each is its own distinct entity. So I might browse Janx's Pathfinder Feats Library or you might browse Morrus' FASA Star Trek Starships. There might be duplication in that some users might choose to do similar things - but that's OK. Choice is good; the better ones will get used more.

Terminology I'm using here: A database contains records, each of which has a number of fields. I don't know if that's the same terminology you use, but it was the terminology used at the NHS database I used when I worked part time at a hospital 25 years ago, and the library database I used at my local library, and the terminology used by the database program I painstakingly typed out 2E kits and Star Trek spaceships in many years ago! So Janx's 2E Monster Database contains monsters (records) each of which has a number of fields (hit dice, size, other stats).

So you're right in that they'll be in categories. The categories could be anything; I'll just set up categories as and when they're needed. So the AD&D 2E category might contain a number of folks' databases - your monster one, perhaps, and a spells one I made.

Once you're looking at a specific database, you can search it (obvious) or use filters via little dropdowns which say "X is Y than Z" - so "Hit Dice is smaller than 6" or "Challenge Rating is larger than 12" or "Name contains the word goblin". These are basically =, >, <, contains, does not contain, and so on. A fairly standard set of common operators. Right now I'm thinking three filters will be available - so you can filter a database by up to three things. Or just search it, of course.

Each database is owned by a person. That person can set permissions for it. Permissions to read, to write, etc. They might even be able to charge for access to it (which would rarely be a good idea - but some smaller companies/IP holders might like to use it for a DDI compendium style thing without having to code their own one).

 

[Janx]

Kinda. Each is its own distinct entity. So I might browse Janx's Pathfinder Feats Library or you might browse Morrus' FASA Star Trek Starships. There might be duplication in that some users might choose to do similar things - but that's OK. Choice is good; the better ones will get used more.

I figured something like that. Keep in mind, it's not Janx's PF Feats Library. It's Janx's PF Library with Skills, Feats, Monsters, Spells, NPCs inside of it. Hierarchy is important, to organizing data, and recognizing the containing units is part of that.

To form a tree representing the data:
Owner: Janx
--GameSystem: PF
----Category: Skills
------Fishing
------Cooking
------Rope Use
----Category: Monsters
------Orc
------Goblin
----Category: Feats
----Category: Spells
----Category: Recipes (or some other crazy thing that nobody else would use this for)
etc.

Terminology I'm using here: A database contains records, each of which has a number of fields. I don't know if that's the same terminology you use, but it was the terminology used at the NHS database I used when I worked part time at a hospital 25 years ago, and the library database I used at my local library, and the terminology used by the database program I painstakingly typed out 2E kits and Star Trek spaceships in many years ago! So Janx's 2E Monster Database contains monsters (records) each of which has a number of fields (hit dice, size, other stats).

This is where professional terminology and normal person terminology collide. What you describe as records with fields, is called a Table. A database is a collection of Tables, many of which are related to each other. A simple rule of thumb, is if you can manage it in one simple sheet in Excel as a list of things with columns describing those things, that's a Table.

So all the Tables the users ever make is what constitute a Database and would be stored in Repository (SQL Server, MySQL, PostGRES, Oracle, etc). Other kinds of Repositories could be used (object database, document database, flat files, XML, etc), but usually it's something SQL based.

So you're right in that they'll be in categories. The categories could be anything; I'll just set up categories as and when they're needed. So the AD&D 2E category might contain a number of folks' databases - your monster one, perhaps, and a spells one I made.

One of the challenges you're going to have is that the rule system defines widely varying fields for the same Category. So the fields needed to describe 2E spells is different than the fields needed to describe 4E spells, even though there is some overlap.

So, the system design may require us to call them the same thing (Spells), but the Table structure needed to store 2E spells will be different than what's needed for 4E spells.

I spent 3 weeks EXPLAINING a design for a system that can handle this to a group of developers who were going to build it. Your guy may be able to skip that step as it's "just him" but tread lightly, dynamic database handling is not simple. Mayhap your guy is using something that's done this already (I've rolled my own 3-4 times), but if he is doing it from scratch, it will take more than a week to complete.

Once you're looking at a specific database, you can search it (obvious) or use filters via little dropdowns which say "X is Y than Z" - so "Hit Dice is smaller than 6" or "Challenge Rating is larger than 12" or "Name contains the word goblin". These are basically =, >, <, contains, does not contain, and so on. A fairly standard set of common operators. Right now I'm thinking three filters will be available - so you can filter a database by up to three things. Or just search it, of course.

Yup. This part is actually the easy part. That GridView product I linked to (again, not a sales pitch), incorporates filter/searching it into it, so I get it for free (well, I did pay $1000/year for the license). There are nice open source solutions as well (jQuery is free and kicks butt!).

Each database is owned by a person. That person can set permissions for it. Permissions to read, to write, etc. They might even be able to charge for access to it (which would rarely be a good idea - but some smaller companies/IP holders might like to use it for a DDI compendium style thing without having to code their own one).

Probably one of the concerns to have is if I upload all my 2E data (which is NOT OGL), and make that public, I am possibly violating copyright. Might be good to include a "report suspected violation" button.


Here's another wrinkle in the design (meaning, do or do not build this, but know why):
Relationships between Tables

Skills, Feats, Spells are what we call Lookup data or Meta Data. They form lists that get associated with other things, but often aren't meaningful on their own (you might look up Swimming to see the rule, but you're really doing it because the Monster you are looking at has the Swimming skill).

In D&D, the most common "other thing" to be associated to is Monsters.

A Monster has zero or more Spells available to it.
A Monster has zero or more Feats assigned to it.
A Monster has zero or more Skills assigned to it.

Thus, when I view the Orc monster, I would see a Feats section, which is a list of the Feats the monster has. These are actually hyperlinks to their respective Feat detail block (pop-up, tooltip, whatever).

This concept of Relationships is where a Database really forms in the modern parlance. It's not manageable with just a single worksheet in Excel.

Here's the good news, you do NOT have to design for this. I am simply raising it to your attention that it is possible, and it makes sense to do (later).

Instead, when you define the 3e Monster Table with Name, HD, AC, Attacks, Damage, etc for fields, you would put in a field called Feats that is a big text box (remember each field should have a datatype, as in a restriction on what kind of data goes in it, and thus what to show the user for data entry/formatting).

When I am entering my new Monster for "Orc", when I get to the Feats section, I can just type in the names of the feats. And that's what the user will see. No fancy hyperlinks, etc.

Or you could let me paste in hyperlinks from the proper Detail screen for the feats (like we do in the forums).

This is basically what I do in my spreadsheet for managing my PC. I just list my feats in cells, and then hyperlink them to the SRD myself. A simple solution.

Just know that a true Database would handle it Relationally, and you'd get more powerful behavior (at the cost of added complexity).

 

[Morrus]

Ah, no, I haven't explained it well. You're describing a very different product there.

I'll have another try a bit later. This is where I wish I had screenshots to show.

 

[Janx]

Ah, no, I haven't explained it well. You're describing a very different product there.

Nothing is ever simple. Normal people do not understand or think about the shape of data and what it takes to actually model it in a technologically useful way.

Here's a link to the site where somebody made a d20 database in a variety of formats (Access, SQL, MySQL, etc):
http://www.andargor.com/

One of the handy things is when you DO build your system, I can easily extract exports from it to help populate your system with this d20 content.

This is the table definition for a Feat in d20 v3.5:
CREATE TABLE spell (
id INTEGER,
[name] VARCHAR(100),
altname VARCHAR(255),
school VARCHAR(255),
subschool VARCHAR(255),
descriptor VARCHAR(255),
spellcraft_dc VARCHAR(255),
[level] VARCHAR(255),
components TEXT,
casting_time VARCHAR(255),
range VARCHAR(255),
target VARCHAR(255),
area VARCHAR(255),
effect VARCHAR(255),
duration VARCHAR(255),
saving_throw VARCHAR(255),
spell_resistance VARCHAR(255),
short_description VARCHAR(255),
to_develop TEXT,
material_components TEXT,
arcane_material_components VARCHAR(255),
focus TEXT,
description TEXT,
xp_cost TEXT,
arcane_focus VARCHAR(255),
wizard_focus VARCHAR(255),
verbal_components VARCHAR(255),
sorcerer_focus VARCHAR(255),
bard_focus VARCHAR(255),
cleric_focus VARCHAR(255),
druid_focus VARCHAR(255),
full_text TEXT,
reference VARCHAR(255)
)

The bottom half has some fiddly bits so he can define what classes can access the spell, but the top half should be familiar and kind of obvious.

Now, will these fields apply to PathFinder's spells? D&D 4E, GURPS, ShadowRun, D&D Next?

Some probably, like the name and description. But the rest is stricly fields defined by the structure of spells in D&D 3.x.

That means in practical terms, you can't define a generic Database for Spells, and stick everybody's spells in there (well, you could one design pattern is to have ALL the unique fields people need, and then hide the ones they don't by keeping a definition somewhere of what fields a ruleset needs).

What I'm saying here is, if you haven't realized that Spells are described very differently between game systems and will need different fields to contain them, and your developer hasn't figured that out, you will NOT get the product you think you want.

Because I am versed with the subject matter (RPGs) and with database design, I understand the nuances that have to be dealt with.

I caution you that if you think what I'm talking about is out of scope with your product, that you may be making a mistake. Your developer may not realize what you're really asking for, and will give you a database suitable for holding PathFinder data, but not one suitable for other game systems (even related ones).

 

[Janx]

Ah, no, I haven't explained it well. You're describing a very different product there.

I'll have another try a bit later. This is where I wish I had screenshots to show.

Just to be clear, that tree example wasn't of a user interface. It was to show the shape of the data internally. A pseudo-entity relationship diagram.

Have your guy post the ERD to the database design he plans to use. If he doesn't know what that is, fire him.

 

[Morrus]

That means in practical terms, you can't define a generic Database for Spells, and stick everybody's spells in there

No, no! That's not remotely what I'm getting at. *I'm* not defining what any database looks like. Everyone defines their own databases. I might set up my 3E spells database differently to your 2E spells database differently to my PF classes database different to your Lego collection database. The system doesn't need to know what a spell looks like - the user does!

I've not managed to communicate what this thing is!

So I create a database called "Morrus' Pathfinder Spells Database". I tell it it has the fields Name, Components, Range, Duration, etc. When I add a spell to that database, I get presented with a form to fill out - which is those fields I defined.

You create a database called Janx's Shadowrun Spells Library. You tell it it has the fields -- err, well, whatever Shadowrun has. When you add a spell to that database, you get presented with a form to fill out - which is those fields you defined.

You give each field a name and a type. So my 3E spells database might have:

Name: (text field)
Level: (dropdown menu 1-9)
Descirption: (large text field)
etc. etc.

Your Shadowrun spells database would look very different, depending how you chose to set it up.

(Of course, if you import an xls file or something, that will decide your fields for you).

 

[Janx]

No, no! That's not remotely what I'm getting at. *I'm* not defining what any database looks like. Everyone defines their own databases. I might set up my 3E spells database differently to your 2E spells database differently to my PF classes database different to your Lego collection database. The system doesn't need to know what a spell looks like - the user does!

But that;'s EXACTLY what I am describing.

When I make MY database for MY spells in the EN system that's tables and structure that has to be defined and stored.

When you make YOUR database for YOUR spells in the EN system, that's tables and structure that has to be defined and stored.

We inherently have different structures, but they all need to be stored within your Repository (probably MySQL or PostGRES).

This is called dynamic data definition. It is on the fly generation of structures by user input to store data flexibly.

It is not trivial, and thus requires real work to make.

 

[Morrus]

Well then yes, that's what I want to make.