Boffins found a role for AI chatbots where habitual hallucinations aren’t necessarily a liability.

Eggheads – based at the University of Pennsylvania and the University of Maryland, Baltimore County, USA – turned to OpenAI’s large language models (LLMs) to help them in fantasy role-playing games, especially Dungeons & Dragons (D&D).

In a preprinted paper entitled “CALYPSO: LLMs as Dungeon Masters’ Assistants”, Andrew Zhu, PhD student at UPenn; Lara Martin, assistant professor at UMBC; Andrew Head, assistant professor at UPenn; and Chris Callison-Burch, associate professor at UPenn, explain how they used LLMs to improve a game that relies heavily on human interaction.

D&D first appeared in 1974 as a role-playing game (RPG) in which players assumed the role of adventurous medieval heroes and played these personalities under a storyline led by a dungeon master (DM) or game master (GM). The prerequisites were a set of rules – published at the time by Tactical Studies Rules – polyhedral dice, pencil, paper and a shared commitment to interactive storytelling and modest theatrics. Snacks, technically optional, must be taken into account.

Alongside these tabletop role-playing games, the proliferation of personal computers in the 1980s led to various computerized versions, both in terms of computer-assisted play and fully electronic simulations – such as the recent Baldur’s Gate 3to name just one of hundreds of titles inspired by D&D and other RPGs.

Academic players from UPenn and UMBC sought to see how LLMs could support human DMs, who are tasked with setting the stage where the mutually imagined adventure unfolds, rolling the dice that determine the outcome of certain actions, to enforce the rules (which have become quite extensive), and to generally ensure that the experience is fun and entertaining.

To do this, they created a set of three LLM-powered interfaces called CALYPSO – which stands for Collaborative Assistant for Lore and Yielding Plot Synthesis Objectives. It’s designed for playing D&D online via Discord, the popular chat service.

“When given access to CALYPSO, DMs reported that it generated high-fidelity text suitable for direct presentation to players, and low-fidelity ideas that the DM could develop further while retaining their creative agency,” explains the newspaper. “We view CALYPSO as an example of a paradigm of AI-augmented tools that provide synchronous creative assistance in established game worlds and tabletop games more broadly.”

The COVID-19 pandemic has moved some in-person online tabletop games, the researchers observe in their paper, and many gamers who play via Discord do so with Avrae – a Discord bot designed by Andrew Zhu, PhD student at UPenn and co-author of the CALYPSO article.

“The basic ideas of the article (that LLMs are able to act as co-DMs in ways that inspire the human DM without taking creative control of the game) apply to D&D and other tabletop games , regardless of modality. But there are still some challenges to overcome before applying the technology to in-person games,” Zhu said in an email to The register.

Zhu and his colleagues focused on the Discord play-by-post (PBP) game for several reasons. First, “Discord-based PBP is already text-based, so we don’t have to spend time transcribing speech to text for an LLM,” he explained.

The online setup also allows the DM to view the output generated by LLM privately (where “low fidelity ideas” matter less) and it saves the DM from having to type or dictate into an interface.

CALYPSO, a Discord bot with source code, is described in the article as having three interfaces: one for generating configuration text describing an encounter (GPT-3); one for targeted brainstorming, in which the DM can ask the LLM questions about a meeting or refine a meeting summary (ChatGPT); and one for open-domain chat, in which players can engage directly with ChatGPT acting as a D&D-savvy fantasy creature.

Image of CALYPSO bot output (click to enlarge)

Configuring these interfaces involved seeding the LLM with specific prompts (detailed in the article) that explain how the chatbot should respond in each interface role. No specific model training was required to integrate D&D operation.

“We found that even without training, the GPT model series knows a lot about D&D by having seen source books and internet discussions in its training data,” Zhu said.

We found that even without training, the GPT model series knew a lot about D&D from having seen source books and internet discussions.

Zhu and his colleagues tested CALYPSO with 71 players and DMs, then asked them about the experience. They found the AI ​​assistant helpful most often.

But there was room for improvement. For example, in a meetup, CALYPSO simply paraphrased the information in the settings and stats prompt, which the DMs said added no value.

The register asked Zhu if the tendency of LLMs to “hallucinate” — make things up — was a problem for study participants.

“In a creative context, this becomes a little less meaningful – for example, D&D reference books don’t have all the details about every monster, so if an LLM says a certain monster has certain colored fur, that matters- like a hallucination?” Zhu said.

“To answer the question directly, yes; the model often ‘makes up’ facts about the monsters that aren’t in the source books. Most of them are trivial things that actually help the DM, like the sound of the call of a monster or the shape of a monster’s iris or things like that Sometimes, less often, it hallucinates more dramatic facts, like saying that frost salamanders have wings (they don’t have none).

Another issue that arose was that model training guarantees sometimes interfered with CALYPSO’s ability to discuss issues that would be appropriate in a D&D game – such as race and gameplay.

“For example, the model sometimes refused to suggest (fantasy) races, likely due to efforts to reduce the potential for real-world racial bias,” the document observes. “In another instance, the model insists that he is unable to play D&D, likely due to efforts to prevent the model from claiming abilities he does not possess.”

(Yes, we’re sure some of us have been there before, denying any knowledge of RPGs despite years of playing.)

Zhu said it was clear people didn’t want an AI DM, but were more willing to allow DMs to rely on AI help.

“During our formative studies, a common theme was that people didn’t want a standalone AI DM, for several reasons,” he explained. “First, many players we interviewed had played with tools like AI Dungeon before and knew about AI weaknesses in long-context storytelling. Second, and more importantly, they said that having an AI DM standalone would take the spirit out of the game; since D&D is a creative storytelling game at heart, having an AI to generate this story would seem wrong.

“The fact that CALYPSO was an optional thing that DMs could choose to use as much or as little as they wanted helped keep the creative ball in the human DM’s court; often what happened was that CALYPSO gave the DM just enough of a boost to break once the human DM felt they wanted more control over the stage, they could just continue DMing in their own style without using CALYPSO at all.” ®