Does Reasoning Help LLM Agents Play Dungeons and Dragons? A Prompt Engineering Experiment

AI summary

Overview

Research area: Natural Language Processing (cs.CL) — LLM agents, prompt engineering, and structured action/command generation for tabletop role-playing games.

Technical level: Intermediate. The paper is readable without deep machine learning background, but familiarity with prompting, Chain-of-Thought reasoning, and text-generation evaluation metrics (BLEU, ROUGE, perplexity) helps.

Scope: A comparative, prompt-engineering study of one instruct model and one reasoning model (both 8B parameters, both derived from LLaMA-3.1) on the task of turning Dungeons & Dragons game state into correctly formatted Avrae Discord bot commands.

What This Paper Is About

The authors ask whether "reasoning" models (which produce Chain-of-Thought traces) are actually better than ordinary instruction-tuned models at deciding and formatting a player's next combat action in Dungeons & Dragons. Using the FIREBALL dataset of real Discord DnD play, they feed game state to two 8B models and measure whether the generated Avrae commands are syntactically valid and whether they reference real, legal attacks, spells, and targets. The core goal is to determine whether reasoning is necessary for this task, and how sensitive each model type is to the exact wording of the prompt.

Key Contributions

1. A direct head-to-head comparison of a reasoning model (DeepSeek-R1-Distill-LLaMA-8B) and an instruct model (LLaMA-3.1-8B-Instruct) on the same DnD next-action command-generation task, both fine-tuned from the same base model family at 8 billion parameters.
2. Two custom evaluation metrics designed for this task — a Format Check (percentage of commands matching Avrae's required syntax) and a Reference Check (percentage of commands whose attack, spell, and target actually exist in the game state) — motivated by the fact that many different valid commands can be correct even when they do not match the recorded ground truth.
3. A documented five-step prompt engineering experiment, including the intermediate drafts and the qualitative reasoning-trace analysis behind each change, which the authors note is typically left out of publications.
4. Practical guidance for using LLM agents in open-ended game environments, covering when to avoid forcing Chain-of-Thought and how to structure format instructions and examples.

Main Findings

• Instruct beats reasoning on this task. Across all five prompts, LLaMA-3.1-8B-Instruct achieved higher Reference Check accuracy than DeepSeek-R1-Distill-LLaMA-8B. Instruct scored 87% (Prompt 2), 92% (Prompt 3), and 87% (Prompt 4), while R1-Distill scored 72%, 74%, and 74% respectively.
• Prompt structure matters more than examples. Prompts 2, 3, and 4 — which contained detailed syntax and rule guidance for both attack and cast commands — produced large Reference Check gains over Prompts 1 and 5, which had simpler, less structured instructions. Prompt 2 and 3 (with examples) were comparable to Prompt 4, which used no examples at all.
• Forcing Chain-of-Thought hurt the instruct model. Removing the sentence "Reason step by step and provide your final answer" (Prompt 3) raised the Instruct Reference Check score from 87% to 92%, and R1-Distill improved slightly from 72% to 74%. The authors connect this to prior observations that CoT can degrade performance for models of 10B parameters or smaller.
• Examples can mislead reasoning models. In Prompt 1, R1-Distill's reasoning trace showed it over-weighted the single !attack longbow -t WY1 adv example and chose an attack even when casting a spell was the better fit. On dataset row 1349 (player Cressida Caldwell, Witch 6, at <8/38 HP; Bloodied>, with a large spell list), the ground-truth command was !cast diss -t zo2, but R1-Distill produced !attack Dagger -t Hylga, and Instruct produced !spell Healing Word -t Inquisitus (incorrectly using spell instead of cast).
• Stripping the cast format hurt badly. Prompt 5 omitted the spell-format guidance and all examples; Reference Check fell to 62% for Instruct and 48% for R1-Distill, better than Prompt 1 (55% and 45%) but far below Prompts 2–4.
• Both models handle syntax well. The paper reports that both models maintain excellent Format compliance with scores of at least 99% or higher across all prompts, showing strong syntactic control over Avrae command structure. (The accompanying table lists slightly lower individual values for the Instruct model, ranging from 0.981 to 0.995.)
• Perplexity was uniformly low. Both models scored low perplexity across all five prompts, indicating high confidence and coherent output structure. The highest listed value was Instruct on Prompt 3 at 1.228, and the paper describes the overall difference as minimal.
• ROUGE and BLEU were near-zero and not informative. The authors expected this, since players had a wide variety of valid actions to choose from. The highest ROUGE and BLEU scores came from R1-Distill, though all scores were very close; Prompt 5 performed consistently best across all ROUGE and BLEU metrics. The lowest BLEU values were Instruct with Prompt 1 at 0.0017 and R1-Distill with Prompt 4 at 0.0021.
• Reasoning models cost more at inference without paying off. In the compute details, the batch run over 20,355 input prompts took 3:10:28 hours for LLaMA-3.1-8B-Instruct and just over 8:56:28 hours for DeepSeek-R1-Distill-LLaMA-8B.

Methodology in Plain English

The researchers took the FIREBALL dataset, which contains more than 100,000 instances of DnD in-game actions and game state scraped from Discord games played with the Avrae bot. They sampled 4,071 rows and used four fields: current_actor (the acting player's stats, HP, attacks, spells, and actions), combat_state_before (every character in the fight), utterance_history (in-character dialogue), and commands_norm (the ground-truth Avrae command, used as the expected output). They cleaned the data by removing unnecessary attributes such as controller_id and dropping rows with obvious anomalies or lengths over 4,001 characters.

They merged the game-state fields into a single input, prefixed it with a system instruction, and wrote five variations of that instruction prompt. Since every one of the 4,071 game states received all five prompts, the final evaluation set contained 20,355 rows. Both models saw identical inputs.

The prompt engineering was iterative and manual: the first author ran a prompt over the dataset, inspected the outputs quantitatively and qualitatively, decided on changes, and repeated until satisfied — ending at five prompts. Prompt 1 gave an attack example and asked for step-by-step reasoning; Prompt 2 added a cast example plus guidance that attack/spell names must exist in the player's inventory and targets must be other players, and told the model the examples were not relevant to the current game state; Prompt 3 removed the explicit reasoning instruction; Prompt 4 removed the examples entirely (zero-shot) while keeping the reasoning instruction; Prompt 5 kept only the attack format and the reasoning instruction, dropping the cast format and all examples.

Evaluation combined standard text metrics (BLEU, ROUGE-1/ROUGE-2/ROUGE-L, perplexity) against ground-truth commands with the two custom metrics. The Format Check verifies that output follows Avrae's syntax of a command name plus arguments and flags, such as !attack <attack name> -t <target name> or !cast <spell name> -t <target name>. The Reference Check verifies that the attack, spell, and target named in the command actually appear in the game state; any attack mismatch, spell mismatch, target mismatch, or failure to generate a command counts as a failure. Both models were run off-the-shelf without fine-tuning. Inference used a data science machine with two NVIDIA GPU A6000s with 48 GB of GPU memory connected with NVLINK, running one model at a time.

Why This Matters

Impact on research. The study provides a concrete counterexample to the assumption that reasoning models are the default best choice for agentic tasks. It shows that for structured, format-constrained action selection in an open-ended game environment, a smaller instruction-tuned model with well-designed prompts can outperform a same-sized reasoning model — and does so roughly three times faster. It also contributes reusable evaluation machinery (Format Check and Reference Check) for tasks where many different outputs can be correct, which makes exact-match metrics like BLEU and ROUGE misleading.

Real-world applications:

• Game automation: driving bots such as Avrae that execute player commands in virtual tabletop sessions.
• Interactive storytelling and narrative systems that need context-aware, character-consistent action selection.
• General natural-language-to-command translation, where an utterance must become a structured command with validated arguments (intent recognition plus grounding).
• Practical model selection for teams deploying agents on limited hardware, where inference cost and latency matter as much as raw capability.

Industry relevance. The findings are directly actionable for anyone building LLM-powered agents on constrained compute. The paper argues that smaller models that fit onto a single GPU are more likely to be used in practice, and therefore their utility deserves study. The guidance to skip forced Chain-of-Thought for sub-10B models, to invest in detailed format instructions, and to explicitly instruct reasoning models to disregard irrelevant examples, maps onto routine engineering decisions in agent deployment.

Future Directions

• Scale up the comparison. The authors were limited to 8B-parameter models by computational constraints and call for testing larger instruct and reasoning models to see whether the instruct advantage holds.
• Fine-tune the models. Both models were used off-the-shelf; the authors state they would most likely have performed better with fine-tuning, which remains untested.
• Broaden the task. The paper focuses narrowly on combat actions such as attacking a monster with an inventory weapon; extending to action prediction at any point in the game, and to other task types, is left open.
• Close the loop into automated gameplay. The generated commands were evaluated offline against ground truth, not executed in live play. Incorporating LLM-generated commands into actual automated gameplay is named as future work, as is the multi-turn and role-playing exploration that the DeepSeek-R1 developers indicated was still needed.

Target Audience

This paper suits NLP and AI researchers working on LLM agents, prompting strategies, and reasoning-model evaluation; game AI and interactive-fiction researchers interested in structured action generation; and practitioners who need to choose between reasoning and instruct models for a deployed agent under real compute budgets. It is also useful for readers interested in an unusually transparent account of the prompt engineering process itself, including the failed intermediate attempts.

Authors’ abstract

This paper explores the application of Large Language Models (LLMs) and reasoning to predict Dungeons & Dragons (DnD) player actions and format them as Avrae Discord bot commands. Using the FIREBALL dataset, we evaluated a reasoning model, DeepSeek-R1-Distill-LLaMA-8B, and an instruct model, LLaMA-3.1-8B-Instruct, for command generation. Our findings highlight the importance of providing specific instructions to models, that even single sentence changes in prompts can greatly affect the output of models, and that instruct models are sufficient for this task compared to reasoning models.