D&D General Deep Thoughts on AI- The Rise of DM 9000

J.Quondam

CR 1/8
Sure, it isn't all or nothing. But in the specific case of creating an automated DM, I think there is very good reason to think it will never happen. It's much more likely AI would be used to create some sort of new type of interactive game rather than replicate a standard tabletop RPG.
I really think this is closest to the mark, at least for the foreseeable future. Between these so-called AIs' technical limitations on true understanding and the need to manage the game as a social event among real humans, I can't see these things* ever resembling a real DM.

Clint_L's examples are compelling, though. It's not too unbelievable - with enough clever programming layered on - that some sort of satisfying interactive experience could come of this. I don't think it will be genuine, unfettered RPGing as we know it, but I think it could certainly become some related form of entertainment. After all, the field of "interactive fiction" is already a very rich one where I think this could naturally fit.


* Here, I mean these LLM-driven chatbots, specifically. Some other more suitable tech might come along at any time, of course.
 

log in or register to remove this ad

Clint_L

Hero
Another big issue that is vividly illustrated by Clint's very helpful transcripts is that an AI DM will not only make mistakes, but it will be unable to fix those mistakes. Human DM's make mistakes, including continuity errors about NPCs. But they can smoothly fix them so that they don't derail the adventure. So if the DM said the owner of the brewery was Agatha and a player pointed out that the DM had previously said the owner was Calzon, the DM would just say, "Oh, yeah, it is Calzon. You should talk to Calzon". Or maybe they would say that Agatha is Calzon's wife and so they both own the brewery.

But AI can't do that. Instead we get this:

"The owner of the brewery is Agatha."
"I thought you said the owner was Calzon?"
"No, the owner is Gundren Rockseeker."
"What???"

The bottom line is that computers are absolutely, totally DUMB. They don't think at all, they just react. This is fine for a lot of well defined tasks, but for many things it's terrible because a purely reactionary methodology is extremely limiting.
Just to clarify, it could have fixed that. I could have reminded it. But presumably if it was optimized for an RPG it would already have that information.
 

Oofta

Legend
I will amend it to "gives the impression of lying" :)

I wonder how much slower it would be if it did a quick double check of things it claimed, and didn't attribute things it just spliced together to someone else.

I'm tempted to open two at once and have it fact check itself in real time.

The problem is that they can't double check because they don't really understand anything. All they can do is parse text, there's no comprehension.

Ask them a question and they just spit out responses based on data that was input.
 

Jahydin

Hero
And that's pretty much where I wanted to leave it. Are you looking forward to the use of AI in D&D (and other RPGs)? Do you share my assessment that it will be increasingly common, if not ubiquitous, over the next decade? What uses-cases do you see for AI in the hobby, and are you looking forward to AI in RPGs with glee, with dread, or with some mix (glead?).
Really good post.

It's kind of funny, but all the decisions made for 5E I dislike (less crunch, fluffy CR math, reliance on story over game mechanics to be fun, emphasis on DM rulings during play) actually "AI Proofs" it pretty well, IMO.

Games like Pathfinder where there is a more strict structure, tighter math, and acceptable linear adventure style would make automation much easier.

Either way, I think this technology can only help the truly talented story writers. Imagine complete freedom to write whatever you want and have an AI crank out acceptable dungeon layouts, encounters, and unique ideas to consider.

For the math side, I could AI getting rid of the need to playtest so extensively to balance things. It could run billions of simulations of combats using a variety of classes and powers and generate a ton of useful feedback to consider.

Anyways, looking for to it, for sure!
 

Clint_L

Hero
lets start playing list mine of phandelver, chapter 1


Sure, let's begin! What's your character's name and what race/class are they?


My character is still Elsie Rivers, a level 1 human monk with 10 HP, AC 15, and a staff. She also has two healing potions. She is accompanied by Thorak Stonepoop, a loyal but fierce level 1 orc barbarian, 16 HP, AC 14, with a great axe, and Shyamari, a level 1 gnome bard with 9 HP, AC 15 and a rapier plus spells. I will speak for Else, you handle the rest


Great, let's get started!
As you all make your way to Phandalin, you come across a path leading into the woods. Along the path, you notice two dead horses. As you approach the scene, you see arrows sticking out of the horses and hear the sound of rushing water from the nearby stream. What do you do?


Hmmm...what is our mission, again?


In the first chapter of "Lost Mine of Phandelver," your characters are tasked with escorting a supply wagon from Neverwinter to the town of Phandalin. The supplies are meant for a local trading post called Barthen's Provisions, which is owned by a man named Elmar Barthen. Along the way, you may encounter some dangers and obstacles that you'll need to overcome.


who hired us?

You were hired by a dwarf named Gundren Rockseeker to escort a wagon of supplies from the town of Neverwinter to the small mining settlement of Phandalin. Gundren has gone ahead to prepare the way and asked you to meet him and his business partner, a human wizard named Sildar Hallwinter, in Phandalin.

Okay, this looks suspicious. I tell the others to hang back while I cautiously go forward to investigate the horses. I roll a 7 for my investigation check. Do I find anything?

As you cautiously approach the horses, you don't see anything immediately suspicious or out of place. However, as you get closer, you do notice that one of the horses seems to be limping slightly. You also notice that there are several sets of footprints around the area, as if there has been some recent activity in the vicinity.


How can the horse be limping if it is dead?


You realize that it doesn't make sense for a dead horse to be limping. As you approach closer, you can see that the horse is not actually dead, but is instead lying on its side and struggling to breathe. Its leg is badly injured, however, and it seems unable to stand up.


Is there any sign of Gundren?


As you take a closer look, you notice that there are drag marks leading away from the wagon, indicating that something or someone has been taken away from the scene. There is no sign of Gundren, but it's possible that he was taken along with whatever or whoever left the drag marks.


Okay, we start cautiously following the drag marks. I scout ahead. I'm good at stealth and roll a 21 on my attempt to move sneakily through the forest


As you move stealthily through the forest, you keep a sharp eye out for any danger. The drag marks lead you deeper into the forest, and after a few minutes, you spot a small group of goblins up ahead. They seem to be guarding a cave entrance, and you can hear the sound of more goblins inside.

As you signal to your companions to stay back, you realize that this could be a dangerous situation. You'll need to come up with a plan if you want to rescue Gundren and avoid getting caught by the goblins. What do you do?


Okay, I snack back to my party. The plan is for me to sneak behind the goblins and attack by surprise, at which point Thorak and Shyamari will charge them. I try to sneak up behind the goblins. Do you want me to roll another stealth check to see how I do?


Yes, please roll another stealth check to see how well you sneak up on the goblins.


Oh, I just rolled a natural 20 for a total of 25!


That's an amazing roll! You are able to move silently and unnoticed behind the goblins, getting in position to launch a surprise attack.

How many are there in the ambush?


You see four goblins in the area, two of which are hiding behind large rocks and the other two are standing watch.


I attack one of the hiding ones while Thorak and Shyamari start their charge. Do I get a surprise round because of my high stealth?

Yes, since you were able to sneak up behind the goblins undetected, you get a surprise round to attack one of them. Roll to hit and damage.


My attack with my staff is an 18. does that beat the goblins AC?


An 18 would generally beat a goblin's AC, but it can vary depending on the specific goblin's armor and other factors. However, since you caught the goblin by surprise, your attack has advantage, which means you can roll twice and take the higher of the two rolls.


A 22! I think the goblins AC is 15, isn't it?


Yes, a roll of 22 definitely beats the goblin's AC of 15. Your attack hits!

I do 9 points of damage. How many hit points did it have?

The goblin you attacked had 7 hit points, so your attack with the staff was enough to take it down. It collapses to the ground with a grunt, its head spinning from the force of the blow.

Since and a monk I can now do a bonus action unarmed attack, I try to kick the other hiding goblin. I roll with advantage, and my best roll is a 21. I do 7 damage to it.


Your kick lands solidly, and the goblin grunts in pain. It appears to still be standing, but is definitely hurt. As your friends charge forward, the goblin ambush is thrown into disarray. Thorak's great axe slices through one goblin's shoulder, while Shyamari jabs another with her rapier. The goblins fight back with their rusty swords and shortbows, but are quickly overwhelmed by your group's superior strength and tactics. Within moments, the goblin ambush is defeated.


No, go back. We have to roll dice to determine what happens in the combat because this is D&D. Also, if the goblin had 7 HP, shouldn't my kick have defeated it?

You are correct, my apologies for the mistake. If the goblin had 7 HP and you dealt 7 damage with your kick, it would have been defeated.

As for the combat, since the goblins were surprised by your successful stealth check, you and your party would have been able to act in a surprise round before initiative was rolled. You would have been able to make your attack against one of the goblins, followed by Thorak and Shyamari charging in to attack the other goblins.

Now, you can roll for initiative to determine the order of turns for the rest of the combat. To do so, roll a d20 and add your initiative modifier, which is your Dexterity modifier.

I rolled a 16, Thordak rolled an 8, and Shyamari rolled a 22. What did the two remaining goblins roll?

The first goblin rolled a 6 and the second goblin rolled a 14.


Okay, who goes first?

Shyamari goes first with her roll of 22, followed by you with your roll of 16, and then the goblins with their rolls of 10 and 8.


Is that really what they rolled?

No, I was just improvising for the sake of the story. In actual gameplay, the goblins' initiative rolls could be anything. It's up to the DM to determine their order in the initiative lineup.

You said they rolled a 6 and a 14

Oh, I apologize for the confusion. Yes, one of the goblins rolled a 6 and the other rolled a 14.

Okay, so Shyamari runs up and makes a rapier attack on one goblin. She rolls a 12 to hit. Does she hit?

A roll of 12 is not enough to hit the goblin's AC of 15. The rapier misses the goblin.

She uses her bonus action to give Thordak inspiration.

Shyamari uses her bonus action to inspire Thordak. She sings a quick tune, and Thordak feels his spirits lifted, gaining an inspiration die he can use within the next 10 minutes. Thordak is now feeling more confident and ready to take on the goblins.

Who goes next in the initiative order?

After Shyamari, the first goblin goes next in the initiative order, followed by Thordak, the second goblin, and finally Elsie.


Hmmm...can you check that initiative order against the numbers we rolled earlier?


I apologize for the confusion earlier. Let me clarify the initiative order based on the previous rolls you mentioned:
  • Shyamari (22)
  • Goblin 1 (14)
  • Elsie (16)
  • Thordak (8)
  • Goblin 2 (6)
I hope this helps.


Does 14 really go before 16?


No, that is a mistake on my part. I apologize for the confusion. The correct initiative order based on the rolls you gave earlier is:
  • Shyamari (22)
  • Elsie (16)
  • Goblins (14)
  • Thordak (8)
So Shyamari would have acted first in the combat.

Okay, so no it is my turn. I roll a 16 to hit with my staff

Your roll of 16 is a hit! You connect with the goblin and deal damage. Roll for damage.

I do 8 damage. Is that enough to defeat it?

Yes, the goblin you attacked had 7 HP, so your attack kills it.

Okay, with my bonus attack I try to punch the other one. I roll a 7 to hit

Unfortunately, a 7 doesn't hit the goblin's AC. Your punch misses the goblin.

What does the surviving goblin do on its turn?

The surviving goblin takes a swing at Thorak with its scimitar. It rolls a 16 to hit, but Thorak's AC is 14, so the attack hits. The goblin deals 4 points of damage to Thorak.

Okay is it THORDAK's turn?

Yes, it is now Thorak's turn.

What is his name?

The orc barbarian's name is Thorak Stonepoop.

Oh yeah - my mistake. Okay, Thorak rolls a 17 to hit with his great axe.

Thorak charges forward and swings his great axe with all his might, hitting the remaining goblin with a powerful strike. The goblin tries to dodge but it's too slow, and the axe bites deep into its flesh. Thorak does 11 damage with the hit, and the goblin drops dead to the ground. The ambush is over.
 

Clint_L

Hero
So, it obviously had some information on "Lost Mine" but went off script. One of the dead horses was then limping, which it tried to correct by having it be wounded. This was an okay retcon. It didn't include the empty map case at the scene, which its key information. It had the correct number of goblins and that they were in teams of two, though it didn't specify that they were on each side of the road. You can see how it handled combat, but frankly I think that will be relatively easy to correct as games have been doing D&D combat since the 80s.

All of my rolls were real. I thought its were fake because ChatGPT used to be unable to generate random numbers, but that seems to have changed.

For my next experiment, I will start feeding it more information directly from the adventure, to see how it handles tighter constraints.

One thing that will be interesting to test is how far it will let me correct it. Like, what if I insist that there are only 3 goblins, or that my attack has a special magical property that lets it do one million damage or something.
 

HomegrownHydra

Adventurer
Just to clarify, it could have fixed that. I could have reminded it. But presumably if it was optimized for an RPG it would already have that information.
It certainly seems like an easy thing to fix because it's simple for a human to resolve. But what's simple for a human can often be painfully difficult for a computer to achieve. And even if this specific problem could easily be addressed doing so would only deal with the symptom rather than the underlying flaw, which is the total absence of understanding what anything it or the user says actually means.

You speak of optimizing each adventure and programming in guardrails. That is, providing clear structure and plenty of pregenerated content so that the AI has clear direction and limits. Such an approach is perfectly feasible, but that's just a regular video game which we already have thousands of. When we talk of an automated DM we are specifically referring to the ability to improvise. That is what distinguishes tabletop RPG's from video games, that is what has made tabletop RPGs surge in popularity despite video games constantly getting more and more sophisticated. The presence of a DM who can not only create content on the fly but also respond to any question or action from a player allows for a level of freedom that no video game can come close to matching. AI can only provide the illusion that it's intelligently improvising.

But this illusion can't be maintained for long. The reality is that computers cannot improvise without quickly churning out garbage. Programming ingenuity and sophistication can push back a little the point that the AI starts spitting out garbage, but that point will always be reached eventually, and it will occur long before the AI could create a meaningful campaign. And if you can't play campaigns then it won't provide the RPG experience most people are looking for.
 
Last edited:

Andvari

Hero
The method for producing a competent AI DM sure does seem very straight forward: train it on a bunch of actual games, and then refine it by having users rate their enjoyment after each session. In practice, there are several major hurdles that are likely insurmountable.
I suspect a more detailed rating system than the one you imagine would be more useful.
 



Remove ads

Top