LLMs as a GM

[Gorgon Zee]

AI might, LLMs won't.

LLMs require significant work to even manage (c.f. this entire discussion), and ultimately are just predictive text writ large.

They can't think. They can't come up with solutions. They can only spew out words in orders that seem likely to be appropriate to their prompts.

So .. isn't that exactly what I do when I am a GM? I respond using words in orders that seem likely to be appropriate to the statements of my players?

I'm not going to respond to "they can't think" since there is no solid definition of what thinking is. But they certainly can come up with solutions. They do so by manipulating patterns of words so as to create a chain of arguments that is consistent with the patterns it has been trained on. This is different in approach from the way that previous AI attempted the problem. The old way is:

1. Go from a text description to one in a formal language
2. Infer new statements in the formal language based on existing statements
3. Translate those new statements into a usable text form

LLMs don't have that formal language; the vectors they use are really just words in a more computer-friendly format. So for many people, it doesn't feel like real thinking. We (humans) don't depend on text to reason, but reason based on concepts which are linked to words, rather than directly on the words themselves.

But modern chain-of-reasoning LLMs specifically do come up with solutions by planning what they need to do, creating sub-steps and evaluating how well those steps worked, generating new steps based on the previous step's results and deciding when they have enough evidence to present a solution. These are all ways we reason. The big difference is that what the LLM manipulates is bundles of words that can be though of as defining concepts, rather than manipulating concepts directly.

Further, if used in a videogame, LLMs (as opposed to SLMs), must be based on distant servers, and every query going to them will have an associated cost (and it ain't nothing, because they use insane and ever-increasing amounts of power/water/etc.) ... And LLMs are getting more expensive to run, not less, as developers try and improve their performance via brute force methods.

The per-query cost in terms of power and dollars is decreasing pretty rapidly. The new models often require multiple queries to do their reasoning, so the total cost is higher. However, yet newer methods are showing that smaller fine-tuned models can do as well as expensive models.

I am pretty sure, right now, that I could run a fine-tuned LLM on my $600 mac that would do as good a job as general purpose gpt-4. I haven't done that because it would cost me $5000 and the results could not be shared as it incorporates IP that I do not have rights to. But WotC easily could.

Now, to be clear, I'm not bullish on it happening. But it's not because they cannot reason well (they can) or will get more expensive (they will get cheaper), it's because they are not good at surprising you. They are designed to present a sort of average experience. There have been interesting papers showing that for a single person, AIs tend to be more creative than that person, but for a group they tend to be less creative than the group, because that creativity is very similar.

I have used LLMs to suggest scenes and create descriptions, and while initially they look good, after a few examples they start to feel very same-y. In general, I think this will be a challenge n any effort that attempts to use AIs to get creative results. It will start looking cool, but after a while will start feeling too similar to previous results.

I use GenAI in Photoshop. Not to be imaginative, but to fill in missing content (it's really good at fixing edges when adding a person to a group shot ...) and that's what I expect to use it for it RPGs in the future.