AI GMs

[Bill Zebub]

Wow, the visceral reaction I had to that sentence.

Well, let me clarify:
1) There is a lot about GMing that is unique to the hobby about which LLMs may not know anything. Then again, there is a lot of text information out there about the hobby including examples of play and guidance for GMs, so it might.
2) Then there's just basic storytelling skill. So far I'm blown away by how creative ChatGPT has been. It would not surprise me in the least if it were able to give me feedback on my storytelling skills.

I wouldn't recommend judging what it can or cannot do before exploring this personally.

 

[Rune]

It absolutely could. I bet it would even give feedback to help improve GMing.

Don’t know about that last part. There are a whole lot of GM-advice “truisms” on the internet that I would consider an ill-fit for my preferences. And that’s the stuff (presumably) it would be trained on.

 

[zakael19]

I wouldn't recommend judging what it can or cannot do before exploring this personally.

Naw, Im totally comfortable with judging it on this.

 

[Scott Christian]

I uploaded my own ruleset and lore into Gemini and asked it to run a solo adventure using those rules and the provided lore/cultures/monsters. It did pretty well, although there were times it would take liberties with the lore. (It put in a sandspeeder from the Star Wars universe into my medieval style fantasy setting? But that was its most egregious error. The others were very minor.)

One thing I had to tell it to do is make the campaign last only ten sessions - so it needed to wrap up the story by the end of session ten. It did an ok job with this. There were still some loose ends, and there definitely could have been more sessions.

Lastly, I'd like to point out it did a great job with its setting descriptions and roleplay scenarios. The combat was pretty smooth too, although since it had a unique ruleset, it did ask quite a few questions when it was unsure of how a specific ability would work.

I am curious to see what it will be able to do in the future, possibly by adding images, maps, and a more stable memory.

 

[Bill Zebub]

Don’t know about that last part. There are a whole lot of GM-advice “truisms” on the internet that I would consider an ill-fit for my preferences. And that’s the stuff (presumably) it would be trained on.

Oh, sure. There are schools of thought on GMing that on some points are diametrically opposed.

But if you were to prime the pump with specific guidance/principles it would be able to adapt. A one or two page document outlining the kinds of coaching you want it to do could be very effective.

 

[Waller]

Well, let me clarify:
1) There is a lot about GMing that is unique to the hobby about which LLMs may not know anything.

If we're clarifying, LLM's don't know things. An LLM knows nothing. They just guess based on what an algorithm decides is the most likely word next, not the most accurate one. That's why they hallucinate nonsense so much. They guess at everything because they don't have the capacity or functionality to know anything. That's just not how they work.

 

[Bill Zebub]

If we're clarifying, LLM's don't know things. An LLM knows nothing. They just guess based on what an algorithm decides is the most likely word next, not the most accurate one. That's why they hallucinate nonsense so much. They guess at everything because they don't have the capacity or functionality to know anything. That's just not how they work.

Oh Christ.

Ok, if we're getting all pedantic about anthropomorphisms, the LLM doesn't "guess," either. It selects ("samples") from among weighted probabilities using pseudo-random number generation.

And while correct that the LLM doesn't know anything...in a sapient sense...about RPGs, it has been trained on RPG content and that information is factored into the responses.

When I ask for specific information about Shadowdark, it is able to retrieve that information accurately. I think "know" is a pretty useful shorthand for that trick, but if you have something concise and clear and non-anthropomorphic, my circuits are practically humming with anticipation to have that vocabulary input into my memory banks.

 

[Waller]

Ok, if we're getting all pedantic about anthropomorphisms, the LLM doesn't "guess," either. It selects ("samples") from among weighted probabilities using pseudo-random number generation.

Yes, you just described guessing. It guesses.

 

[Bill Zebub]

Yes, you just described guessing. It guesses.

Yeah because if you ask it how it makes those decisions, it knows…

 

[Heilemann]

Yes, you just described guessing. It guesses.

Unlike people who are of course known to never hallucinate or make cognitive mistakes, or, you know, have a train of thought which is really just... a series of guess the next token... Calling that 'knowing' is in essence not that far off from what people consider 'knowing' which is in itself a deeply flawed concept.

No one is out here claiming sapience, the question is capability, and LLMs are clearly–to many of us anyway–quite capable.

Anyway, as it happens I've built an app for running games for several AI agent players, with built-in PDF support (and PDF sheets), notes, and dice. It works (until it doesn't). But I'm poking away at it anyway. I ran some Night's Black Agents on my phone last summer while sipping cider in the shade at a farm while my daughter slept and the fam was picking cherries; it was surprisingly fun!

One thing that's actually pretty fun is making characters with the AIs. They're so much better at looking up rules than my players!

I'm still playing with having an AI run a game successfully. I still find it way too easy to see the chinks and the edges of the experience. But it's getting better all the time, and some models are better in my opinion; generally speaking Claude is much better at understanding the premise of an RPG (eg. not presenting a list of options, not taking over control of my character, that sort of stuff) when system prompted properly.