AI GMs

[Cergorach]

I've played around with a couple of different LLMs for this, including a specialized service for this (forgot the name). It was interesting, and I can see why people like it. But... I tend to see that people 'try' it and that's generally a short individual session, not a campaign. When we look at the technical side of LLMs we're looking at context windows. Depending on the LLM, they can be small or large, but even those with large context windows have issue the larger the context window gets. Now, you can work with RAG, and/or compressing previous sessions into shorter summaries. But when I look at how LLMs make summaries of D&D sessions of 4+ hours with four different people talking, they aren't all that great. They confuse people and characters, who did what and why, sometimes add hallucinated events or don't weigh events properly. I really wonder how fast a ChatGPT blows a gasket with a 4+ hour D&D session with a party of four players (with voice to text).

I've also noticed that during a 'conversation' with an LLM, the loose 'consistency', either moving to a longer or shorter prose format, going into a different story telling style, or they keep repeating the same stuff. You can make software layers on top of the LLM that could mitigate that behaviour, but that would become more and more costly, both in compute and response time.

While it was interesting to (role)play with the LLMs in that way, I doubt that would ever replace group play or computer game play. i recently tried the introduction to Solasta, which was imho more fun then playing with the LLM, and that is just one of the many, many CRPGs I want to play over playing with the LLM.

I wonder how much of that preference over LLM is due to me recognizing when the LLM messes up drastically and knowing it's limitations... If a child or someone without much technical computer/LLM knowledge tries it, would they be convinced by the illusion?

 

[Bill Zebub]

While it was interesting to (role)play with the LLMs in that way, I doubt that would ever replace group play or computer game play. i recently tried the introduction to Solasta, which was imho more fun then playing with the LLM, and that is just one of the many, many CRPGs I want to play over playing with the LLM.

I wonder how much of that preference over LLM is due to me recognizing when the LLM messes up drastically and knowing it's limitations... If a child or someone without much technical computer/LLM knowledge tries it, would they be convinced by the illusion?

I agree that we're nowhere close to LLMs replacing human GMs. (But at the rate were are going, I wonder how many years it will be before that changes.). And you have to be able to roll with the inconsistency. Here's one snippet from our last session:

But, you know, playing BG3, Solasta, and even WoW also aren't replacements for real TTRPGs, but they are an adjacent kind of fun. In my opinion, AI-GM'd games are similar, fitting in between video games and TTRPGs. They scratch a similar itch, and fill in yet another need.

And they are getting better practically every month.

 

[cranberry]

I use AI to bounce adventure ideas off of/fill in blanks, but I found the output to feel very same-y if you ask it to make something from scratch.

 

[Cergorach]

But at the rate were are going, I wonder how many years it will be before that changes.

From what I'm seeing on the technical side, we're hitting technical walls that can't be solved due to the nature of LLMs. And that would mean we would need something else AI that's not LLM to get over those walls. But we'll see.

But, you know, playing BG3, Solasta, and even WoW also aren't replacements for real TTRPGs, but they are an adjacent kind of fun. In my opinion, AI-GM'd games are similar, fitting in between video games and TTRPGs. They scratch a similar itch, and fill in yet another need.

BG3 and Solasta I don't see as replacements for pnp RPGs, I see them as better replacements for solo RPing with an LLM (personal preference). And I took two examples that are 3D CRPGs, but there are other CRPGs as well, think of Darkest Dungeon (2D) and something like Citizen Sleeper (text based)...

Something like WoW (which I never could get into) would probably the best replacement for a group based TTRPG, if you played in groups. It probably gives you the most freedom of the computer based solutions while still interacting with fellow players. If you would want something more D&D like, the D&D MMO is pretty darned good (for D&D).

Someone mentioned LLM solo RPing being an on demand 'choose your own adventure', that seems pretty accurate. The length of the book being the finite space of the LLM context window. But there are whole categories on Steam that do the 'choose your own adventure' thing, even text based.

 

[Gorgon Zee]

I use AI to bounce adventure ideas off of/fill in blanks, but I found the output to feel very same-y if you ask it to make something from scratch.

While evaluating creativity is hard, there seems to be a reasonable amount of evidence that an LLM can generate material that is perceived as more creative than that created by an average person for a given domain. However the issue is exactly what you have suggested above -- it's creative the same sort of way every time. So a group of people is more creative than a group of LLMs.

They are also terrible at handling multiple factors at once. So for linear adventures -- dungeon crawls, heists, that sort of thing -- they should be able to do a pretty good job, they really cannot handle a complex plot. I have had some success with prompts for specific scenes ("create one paragraph length outlines of a Pendragon encounter which could test a knight's chastity"). They were pretty similar, but they did the job and I could pick one out and it did save me some time. But when I asked it to summarize the plot of Agatha Christie's Peril at End House (which it did well) and then design a TTRPG adventure around that, it was beyond unusably bad; it had to keep track of 10 characters in both the Christie and derived work, create 5 scenes and around a dozen clues. It was just unable to do that in any way adequately.

And there are no innovations in the pipeline for AI that will improve this. They have run out of data to train on, the context windows are already sufficient to hold the info needed, and RAG or agent approaches don't help. It's not an issue with finding the right material -- I fed it all that. The issue is a fundamental inability to create complex output based on syntactic patterns only.

So, overall, my guess is that AIs can be trained to be as good as an above-average zero-prep GM. If that's the sort of game you like, you should have no issues. Feed it your sandbox world and set a direction and it will build the next next encounter for you one after another -- so long as, like in the original Star Trek TV show, you don't expect any long-term effects of your actions. But if you are looking for a game that has a more complex plot involving many game world factors together, I don't currently see a path for AIs to do that.

 

[Reynard]

We should all remember that LLMs are trained on us. As it relates to D&D, that is probably literal: it's D&D understanding comes from decades of archives of message board content and, more recently, YouTube content.

An AI GM is going to be an aggregate of the internet's ideas of what GMing is.

Which we can assume is focused way too much on the negative, just based on how internet discourse works.

 

[Tigris]

Ah I would give a different explanation. With how much GMs online excuse their misbehaviour and how often GMs form bubbles in online discussions, I would expect that a lot of the bad GM behaviour of an AI just comes from these GM discussion groups. So we as the players must train the AI the often overheared player opinion in order to make the AI GM not fall back on the typical bad GM behaviour like not caring for player fun, ignoring player oppinion and trying to limit player choices.


Still I am hopefull and quite positive that in the future GMs can be replaced by AI GMs, we as players just need to use and train them enough, I am sure AI is more flexible than people who GM (badly) since 30+ years.

 

[Bill Zebub]

BG3 and Solasta I don't see as replacements for pnp RPGs, I see them as better replacements for solo RPing with an LLM (personal preference).

Oh, interesting. I don't enjoy the RPing in BG3 and Solasta at all. I click through the dialog as fast as I can, and mostly just kick open the door, kill the monster, and take its stuff. I know it's all canned dialog and fixed pixels, and therefore the "plots" bore me.

With the LLM I enjoy knowing there aren't defined solutions to problems; that I can color outside the lines as much as I want and try any crazy scheme I can think of. Any blank space I explore, the LLM will create content for me.

Something like WoW (which I never could get into) would probably the best replacement for a group based TTRPG, if you played in groups.

I basically spent all of 2007 raiding full time in WoW. I loved it, and some of those people are still friends. But it is absolutely nothing like playing a TTRPG.

 

[Bill Zebub]

They are also terrible at handling multiple factors at once. So for linear adventures -- dungeon crawls, heists, that sort of thing -- they should be able to do a pretty good job, they really cannot handle a complex plot. I have had some success with prompts for specific scenes ("create one paragraph length outlines of a Pendragon encounter which could test a knight's chastity"). They were pretty similar, but they did the job and I could pick one out and it did save me some time. But when I asked it to summarize the plot of Agatha Christie's Peril at End House (which it did well) and then design a TTRPG adventure around that, it was beyond unusably bad; it had to keep track of 10 characters in both the Christie and derived work, create 5 scenes and around a dozen clues. It was just unable to do that in any way adequately.

I will agree that when I've asked LLMs to generate large scale "plots" the results have been kind of disappointing and vanilla.

But I seem to get much more interesting results when it is GMing, and it only has to describe the next thing I see. I think that's because it's using my responses and actions to generate new content. So my own creativity is, in a sense, seeding its creativity. Iteratively.