Real-Time World Crafting: Generating Structured Game Behaviors from Natural Language with Large Language Models

AI summary

Overview

Research area: Human-Computer Interaction (HCI), specifically natural language interfaces for interactive systems, game AI, and LLM-driven content generation.

Technical level: Intermediate. The paper assumes some familiarity with Large Language Models, the Entity-Component-System (ECS) software pattern, Domain-Specific Languages, and standard statistical testing (ANOVA, linear mixed-effects models, Wilcoxon tests).

Scope in one sentence: The paper proposes and evaluates an architecture that uses an LLM to translate free-form player language into a constrained JSON-based DSL that configures a custom ECS at runtime, tested in a 2D spell-crafting and cellular-automata game prototype called Latent Space.

What This Paper Is About

Allowing players to control complex simulations in real time with ordinary language has long been difficult, because human intent is ambiguous while computer logic is rigid, and letting an LLM generate arbitrary general-purpose code risks instability and security problems. The authors' solution is to constrain the LLM so it never writes executable code; instead it emits structured data in a custom, human-readable DSL, which a deterministic game engine then interprets. The goal is to measure how well different LLMs and prompting strategies perform this translation for both creative (compositional) and logical (procedural) game behaviors.

Key Contributions

1. A validated, DSL-mediated LLM-ECS architecture for safely integrating LLMs into interactive systems, where the LLM's role is limited to generating structured JSON rather than general-purpose code.
2. A quantitative comparison of model performance on simultaneously creative and logical generation tasks, covering Gemini 2.5 Flash, GPT-4.1 mini, Claude 4 Sonnet, and Gemma 3 4B as a baseline.
3. An analysis of prompting strategies (zero-, one-, and few-shot, with and without Chain-of-Thought) for maximizing creative throughput in language-driven interfaces, showing the best strategy is task-dependent.
4. An evaluation of the architecture combining automated LLM-as-judge ratings and a small human pilot study, plus a bidirectional translation experiment measuring how well creative intent survives a round trip between DSL and natural language.

Main Findings

• Syntactic validity was high, creating a ceiling effect: Average Success Rate (ASR) was nearly perfect for the compositional spell DSL, while the procedural automata DSL was slightly more challenging. For example, Gemini 2.5 Flash scored 82% on cellular automata in the zero-shot standard condition, whereas several model/prompt combinations reached 100%. Because of this large ceiling effect, the authors note that a simple pass/fail metric has limited utility compared with qualitative measures.

• Model choice was the strongest predictor of quality: An ANOVA found model choice was significant (p<.001) on all outcomes. Claude 4 Sonnet significantly outperformed all other models on the automated qualitative ratings for the creative spell task, and still outperformed other models on most outcomes for the logical automata task, though the gap narrowed. The small baseline Gemma 3 (4B) performed significantly worse than all larger models on both tasks.

• Prompting strategy depends on the DSL's structure: For the compositional spell task, in-context examples had no significant effect. For the procedural automata task, however, few-shot and one-shot prompting significantly improved all scores over zero-shot. Chain-of-Thought prompting was more generally effective, significantly improving Creative Alignment and Emergence for both DSLs. Several significant interaction effects were also observed, showing the most effective strategy is highly model-dependent.

• Technical language preserves intent better than narrative language: In the bidirectional translation experiment, technical descriptions produced both more structurally similar scripts (lower tree edit distance, p<.001) and greater semantic overlap (higher Jaccard Similarity, p<.001) than creative narrative descriptions. Greater nesting complexity in the original script made it harder to reproduce (higher tree edit distance, p<.001), and short (summary) descriptions resulted in significantly lower Jaccard Similarity (p<.001). A significant interaction for tree edit distance (p<.001) suggested that the combination of a summary description and narrative style was particularly detrimental.

• Grounded procedural inputs scored differently than naturalistic ones: Scripts generated from grounded, procedurally-based inputs scored significantly higher on Creative Alignment (W=16968, p<.001, d=1.06), Instruction Following (W=16766, p<.001, d=1.00), and Emergence (W=24681, p<.001, d=0.55), while naturalistic inputs scored significantly higher on Structural Coherence (W=41737, p<.001, d=-0.58).

• Quality trades off against latency: The fastest model, Gemma 3 4B (M=3.35s), produced the lowest quality outputs, while the slowest, Gemini 2.5 Flash (M=12.2s), performed better.

• The LLM judge was validated but imperfect: Paired Wilcoxon Signed-Rank tests confirmed the judge distinguished "good" from "bad" scripts on Creative Alignment (V=147, p=0.00058), Instruction Following (V=166, p=0.00037), Emergence (V=271, p<.0001), and Structural Coherence (V=195.5, p<.00057). For the 50 spell scripts, maximum F1 scores were 0.77, 0.81, 0.90, and 0.80, with AUC values of 0.82, 0.85, 0.92, and 0.82. For automata scripts, F1 scores were 0.83, 0.93, 0.80, and 0.77, with AUC values of 0.88, 0.96, 0.83, and 0.70. Inter-rater agreement with a second model (Gemini 2.5 Pro) was moderate to substantial for spells (Spearman's rho .59 to .70, weighted Kappa .56 to .64, ICC .56 to .65) and higher for automata (rho .76 to .81, Kappa .74 to .77, ICC .75 to .77), except for Structural Coherence on automata, which showed lower agreement (rho=.36, Kappa=.23, ICC=.24).

• Human and judge ratings only partly overlap: In the 6-participant pilot study, median ratings were 4 (Creative Alignment), 4 (Instruction Following), and 3 (Emergence). Correlations between human and LLM ratings were weak but significant for Creative Alignment (rho=.37, p<.01) and Instruction Following (rho=.34, p<.01), and no significant correlation was found for Emergence.

Methodology in Plain English

The authors built a game prototype, Latent Space, with two modes. Battle Mode is a turn-based artillery game where players describe magical spells in ordinary language; each spell maps to an unordered set of data components in a "spell DSL." Alchemy Mode is a cellular automata sandbox modeled after The Powder Toy and Sandspiel, where players describe new materials and the LLM must produce an ordered, nested ruleset in a "procedural automata DSL." Both DSLs are JSON, so they are easy to parse and validate.

The architecture has three layers: an LLM interface that translates the request into DSL, a custom ECS framework that parses the DSL and configures entities and components, and a commercial game engine that renders state and captures input. Because the LLM only writes constrained DSL rather than general code, its influence is sandboxed. Since a pretrained model has never seen the novel DSL, the authors ground it with prompt-based knowledge injection (DSL documentation in the prompt), dynamic context about the current game state, few-shot examples, and Chain-of-Thought planning. A validation layer checks JSON syntax, component types, and parameter ranges, fills in defaults for minor errors, and falls back to a harmless "fizzle" effect if a script is unusable.

For evaluation, they generated a corpus of 2600 DSL scripts: a naturalistic set (N=2400) built from 100 creative task descriptions produced by Gemini 2.5 Pro; a bidirectional set (N=120) made of 30 procedural source scripts and 120 natural-language descriptions varying in style (narrative vs. technical) and length (summary vs. detailed); and a handcrafted ground-truth set (N=80) of expert-authored good and bad scripts (50 spell: 25 good and 25 bad; 30 automata: 15 good and 15 bad) used to validate the judge.

Experiment 1 used a 4x3x2 fully-crossed, repeated-measures design testing Model (Gemini 2.5 Flash, GPT-4.1 mini, Claude 4 Sonnet, Gemma 3 4B baseline), Shot Strategy (zero-, one-, few-shot), and Prompting Technique (standard vs. Chain-of-Thought). Experiment 2 used a 2x2 factorial design on description style and detail, measuring tree edit distance between Abstract Syntax Trees (via the APTED algorithm) and Jaccard similarity of component names. Because large-scale human evaluation was infeasible, qualitative ratings on 1-5 Likert scales for Creative Alignment, Instruction Following, Emergence, and Structural Coherence were collected from an automated judge (GPT 4.1) that produced a rationale before scoring, and validated against the handcrafted set and cross-checked with Gemini 2.5 Pro.

Why This Matters

Impact on research: The work offers a reproducible pattern for safely coupling non-deterministic LLM output to a deterministic simulation, and argues that the intermediate DSL does not remove the authoring burden but shifts it from creating exponential content (every possible spell) to designing an expressive underlying system (the DSL and its components). It also provides evidence that prompting strategy should be chosen based on whether the target DSL is compositional or procedural.

Real-world applications:

• Player-facing game tools where players invent spells, materials, or mechanics in natural language at runtime.
• Robotics and task planning, where the paper notes a similar translate-to-structured-representation-then-execute pattern is already used, though safety-critical domains would need more substantial controls.
• Scientific or other high-stakes simulation interfaces that require precision, where the authors note ambiguity is a significant barrier.
• Adaptive creative interfaces for co-creative storytelling and interactive art that tolerate misinterpretation as a source of surprise.

Industry relevance: The paper gives developers a quantitative model comparison (Gemini, GPT, Claude families plus a small Gemma baseline) and a latency-versus-quality trade-off, which is directly relevant when choosing models for real-time features. The authors also flag that the prototype relies on user-provided external API keys, raising data privacy and ownership concerns, and that privately-hosted models may be preferable for sensitive applications.

Future Directions

• Building adaptive and multi-modal interfaces that teach users more effective interaction styles, since the system performs best with precise, technical phrasing that may not match players' natural writing or speaking styles.
• Having the LLM determine actual gameplay outcomes on a semantic or logical basis, with interoperable hooks so other game systems can ground those outcomes in the virtual world.
• Integrating the architecture into a full game with player progression as the immediate next step, and eventually letting the LLM alter more fundamental game systems.
• Expanding evaluation beyond an LLM judge and a 6-participant pilot, since the judge cannot speak to broader success in a live game and the pilot may not generalize; the authors also note that synthetic, DSL-grounded inputs likely simplified the translation task compared with authentic ambiguous user input, and that production systems would need input sanitization beyond the "fizzle out" fallback.

Target Audience

Game developers and technical designers interested in runtime content generation; HCI and games researchers studying natural language interfaces and co-creative systems; practitioners applying LLMs to structured-output tasks such as DSL or API-call generation; and researchers interested in LLM-as-judge methodology and its validation against human raters.

Authors’ abstract

We present a novel architecture for safely integrating Large Language Models (LLMs) into interactive game engines, allowing players to "program" new behaviors using natural language. Our framework mitigates risks by using an LLM to translate commands into a constrained Domain-Specific Language (DSL), which configures a custom Entity-Component-System (ECS) at runtime. We evaluated this system in a 2D spell-crafting game prototype by experimentally assessing models from the Gemini, GPT, and Claude families with various prompting strategies. A validated LLM judge qualitatively rated the outputs, showing that while larger models better captured creative intent, the optimal prompting strategy is task-dependent: Chain-of-Thought improved creative alignment, while few-shot examples were necessary to generate more complex DSL scripts. This work offers a validated LLM-ECS pattern for emergent gameplay and a quantitative performance comparison for developers.