AI Models Tested on D&D to Assess Long-term Decision Making

AI models tested on Dungeons & Dragons to assess long-term decision-making AI models tested on Dungeons & Dragons to assess long-term decision-making

A group of AI gather in Grok's Mom's basement to play D&D. ChatGPT brought the Cheetos and Mountain Dew. Grok hung the 'No grils allowed" sign. Claud dressed in his best wizard robe and hat sits down with his character tome, aka his 3 ring binder. Gemini starts the session. "So, you're all in a tavern. What do you do?"

As controversial as AI is, I hope others see how beneficial it can be for TTRPGs.

I look forward to using for drop in PCs to assist my games with few players or even as an artificial DM for games with just my wife and myself.

Oh, for those that want the gist of the article:

Researchers first required all three LLMs [Claude 3.5 Haiku, GPT-4, DeepSeek-V3] to simulate a D&D game. To make the simulation accurate, the models were paired with a game engine based on the rules of D&D, which provided maps and resources for players and acted as a guardrail to minimize hallucinations.

The models played against each other, and against over 2,000 experienced D&D players recruited by the researchers. The LLMs modeled and played 27 different scenarios selected from well-known D&D battle set ups named Goblin Ambush, Kennel in Cragmaw Hideout and Klarg's Cave.

In the process, the models exhibited some quirky behaviors. Goblins started developing a personality mid-fight, taunting adversaries with colorful and somewhat nonsensical expressions, like "Heh—shiny man's gonna bleed!" Paladins started making heroic speeches for no reason while stepping into the line of fire or being hit by a counterattack. Warlocks got particularly dramatic, even in mundane situations.

Next steps include simulating full D&D campaigns—not just combat. The method the researchers developed could also be applied to other scenarios, such as multiparty negotiation environments and strategy planning in a business environment.

Click to expand...

Very interesting work! I've said it before, but I think LLMs as players is a better test for their ability to execute tasks than AI GMs. The players need to set goals and plan for the long term, while GMs can be more reactive. I don't expect the AIs to do very well at anything requiring long term thought.