A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

AI summary

Overview

Research area: AI coding agents and benchmark design for end-to-end software/game generation from written specifications (CS/AI, evaluation methodology).

Technical level: Advanced. The paper combines dependency-graph contracts, weighted scoring, multimodal replay judging, and code-as-policy bot agents.

Scope (one sentence): The paper introduces A2Z GameSpec-Bench, a benchmark of 100 long-form Game Design Documents (GDDs) plus a fixed, dependency-aware evaluation contract that measures how faithfully nine coding agents turn a design document into a complete, playable game across source code, rendered behavior, and live playtesting.

What This Paper Is About

Coding agents can already produce something that looks like a game from a short prompt, but producing the game a designer actually specified is a different problem: requirements in a long GDD interact, so one rule's effects become another rule's preconditions, and these interactions must survive across game logic, visual rendering, and player interaction. Existing game-development benchmarks use substantially compact input specifications and provide limited support for checking interdependent requirements in long-form GDDs through a shared representation. This paper builds a benchmark and an evaluation pipeline whose goal is to measure GDD Fidelity — whether a generated game satisfies the GDD's requirements and preserves the relationships among them.

Key Contributions

1. A benchmark of 100 long-form GDDs for full-game generation. The corpus contains 50 Small designs (average 14,085 tokens) and 50 Big designs (average 26,297 tokens), each built from a game brief through an agentic authoring pipeline that checks for omissions and inconsistencies. Each task requires a source project and a playable build, in a 2D single-player Phaser setting for controlled comparison.

2. A Dependency-Aware Contract as a fixed evaluation target. Each GDD is converted into a contract consisting of rules (conditions, triggering events, expected effects) and invariants (constraints within a stated scope), connected by directed edges that link a rule to another when it updates a state the other's preconditions read, or emits an event that triggers it. The contract is constructed from the GDD, withheld from the building agent during initial generation, and frozen across agents and revision rounds.

3. A three-axis evaluation pipeline driven by agent-generated Test Policies. Source-code inspection checks how requirements are implemented; scenario-based replay collects rendered visual evidence using a fixed input sequence and timing with adaptive frame selection; adaptive playtesting uses code-as-policy bots that select inputs from runtime observations, in both a normal-play mode and a targeted adversarial mode that initializes preconditions for rules normal play did not reach. Judgments and evidence from all three axes stay linked to the same requirements.

4. Dependency-weighted scoring and requirement-level revision feedback. Rules with greater downstream reach receive greater influence on the source-code score (weight floor parameter α = 0.5, giving a rule at most twice the weight of a rule with no downstream dependencies), and invariant violations reduce the score. Overall GDD Fidelity is the equal-weight mean of the source, replay, and adaptive-play scores.

Main Findings

• High compilability and runnability do not imply high GDD Fidelity. Claude-Fable-5.1 achieves a verifiable rate of 98.7% under compile and runtime checks while its overall GDD Fidelity is 77.0. Across the evaluated agents, mean overall GDD Fidelity is 71.1 on Small and 51.1 on Big, while the corresponding verifiable rates are 93.9% and 94.2%.

• Best and worst performing agents. Claude-Fable-5.1 achieves the highest overall GDD Fidelity of 77.0 across all 100 GDDs, followed by Claude-Opus-5 at 73.9; Kimi-K2.7 reports the lowest overall GDD Fidelity in the table at 39.5.

• Bigger specifications are harder on every axis. Overall score decreases by 20.0 points from Small to Big, and the source-code, scenario-based replay, and adaptive playtest scores decrease by 19.1, 20.6, and 20.3 points respectively — the difficulty is not confined to source implementation, visual rendering, or runtime behavior alone.

• Local rule passes hide downstream dependency failures. Across 100 GPT-5.6-Sol games, the mean pass rate for dependency-linked rules is 72.4% from local source-code judgments but 22.7% when all upstream rules must also pass (a graph-based reachability proxy).

• The three axes catch different failures. Among rules that pass with a full source-code score, only 76.8% are confirmed as satisfied during playtesting, while 23.2% do not receive a full score; conversely, 25.7% of rules receiving zero source-code score are judged as satisfied during playtesting. Omitting an axis reverses GDD Fidelity rank orderings across the 100 GPT-5.6-Sol games at rates of 20.5% (source code), 8.9% (replay), and 11.2% (playtest).

• Dependencies redirect inspection to real runtime failures. With builds, enumerated source judgments, and requirement mappings held fixed, adding dependency context increases coverage of recorded playtest violations from 71.1% to 80.2% (a 9.1-point gain, 95% paired game bootstrap interval 6.1–13.0 points). The additional 51 violations span 26 games; among 2,185 targets with matched source-code scores and conclusive playtest verdicts, 560 are violated.

• Dependency context adds the most inspection targets in broad designs. Full-credit targets flagged for further inspection rise from 25.7% on Small to 58.6% on Big, identifying 408 of 908 eligible full-credit targets (44.9%) across 53 games. Violation coverage goes from 16.7% to 24.2% on Small and from 78.3% to 87.7% on Big.

• Requirement-level feedback improves revision more than self-revision. On 50 Big GDDs with GPT-5.6-Sol, the three-axis feedback condition (Source + Replay + Playtest) reaches a mean overall GDD Fidelity of 74.3 after two rounds, exceeding the 67.0 of self-revision by 7.3 points (10.9% relative) from the same initial builds.

• Agents with similar code scores differ sharply at runtime. GPT-5.5 and GLM-5.3 have nearly equal source-code scores (68.5 vs. 68.4) but runtime fidelity differs by 11.4 points (60.6 vs. 49.3); on Big, GLM-5.3 has a higher source-code score (58.1 vs. 55.9) while GPT-5.5 has higher runtime fidelity (52.1 vs. 37.4).

• Game design documents are unusually causal. Across three game-design corpora with 353 documents, CiRA classifies 38.4% of sentences as causal on average, compared with 28% for general documents; the breakdown is 37.9% for studio GDDs, 32.0% for game benchmark specs, and 45.3% for Top-100 BGG rulebooks.

• Concrete cross-axis failures. In Siege Deck 2D, the deck-limit requirement (at 30 cards, remove one before any card gain) receives full source-code credit, but playtesting reveals a reward gain without the card removal, caused by missing state linkage before reward gain. In Chalk Escape, code for the start-zone pulse produces four peaks instead of the specified two; in Neon Spray, the "SETTINGS" text is clipped in the pause menu; in Mobile Balance, the hanging weight keeps following the pointer after release.

Methodology in Plain English

1. Write long design documents. Starting from game briefs, the authors expand each into a creative vision and then a detailed GDD, passing them through an agentic authoring pipeline that checks completeness and consistency. They then split the documents by implementation scope into 50 Small and 50 Big designs.

2. Turn each GDD into a fixed checklist with a dependency graph. Rather than a flat rubric, each document becomes a contract of rules and invariants plus edges saying which rule feeds which. Contract construction fixes a shared state vocabulary, extracts read/write facts to build the dependency skeleton, and fills in conditions, triggers, and effects. The contract is accepted and frozen before any generated build is compared, and the building agent never sees it during initial generation. On average each GDD specifies 69 outcome requirements and 32 invariants, and evaluation uses 20 replay scenarios.

3. Check the game three ways. (a) An evaluator agent reads the source code and rates each rule's implementation completeness from 0 to 1, with invariant checks scored 0 or 1. (b) For replay, a fixed policy of player inputs and timings is executed per scenario, and a judge retrieves relevant frames from the recorded replay to score visible outcomes. (c) For adaptive playtesting, an evaluator-side coding agent writes an executable bot ("code-as-policy") that chooses inputs based on the current game state; normal play runs from the default state with only player inputs, and adversarial tests initialize a state satisfying a target rule's preconditions, then still require player inputs through the same interface, with assisted attempts marked in the trace.

4. Judge and combine. A separate evaluator compares each rule's conditions, trigger, and expected effects against the recorded trace, producing satisfied, violated, or unverified, with three judgments per rule and a strict majority deciding the verdict. Judgment coverage is reported separately. Overall GDD Fidelity is the mean of the source, replay, and adaptive-play scores, and runtime fidelity is the mean of replay and adaptive-play.

5. Feed results back and re-measure. Requirement-level judgments and their evidence (code, frames, traces) become feedback for the next revision round on the same build, and revised builds are scored against the same contract. Evaluator-side agents use GPT-5.6-Luna at high reasoning effort; each configuration generates three builds per GDD and each build is independently judged three times.

Why This Matters

The paper makes a structural point about evaluating generative coding agents: judgments made only from source code can award full credit to implementations that violate the design at runtime, and requirement lists without dependency structure overstate how much of a specification is actually working. Its impact on research is to reframe specification-following as preserving both rules and the relationships among them, and to introduce long-form, dependency-aware, cross-channel evaluation as a benchmark target — the comparator table contrasts this against benchmarks such as GameDevBench, OpenGame-Bench, GameCraft-Bench, PlaytestArena, GameGen-Verifier, and GameXpert-Bench, which differ in input type, relations, and evaluation channels.

Real-world applications:

• Game QA and automated playtesting. The pipeline's combination of scenario replay and code-as-policy bots maps onto the studio practices of code-level checks, visual testing of specific scenarios, and QA playtesting that the paper cites.
• Specification-driven outsourcing of implementation. The setting targets the studio practice of delegating implementation to agents while retaining design control, which requires that the delivered game match the GDD rather than merely resemble a game.
• Agent selection and procurement. Because agents with nearly equal source-code scores (e.g., 68.5 vs. 68.4) can differ by 11.4 points in runtime fidelity, the benchmark supports comparing candidate coding agents on end-to-end behavior.
• Revision tooling. Requirement-linked evidence and dependency-guided inspection direct agents to the prerequisites that must be fixed first, which the paper demonstrates increases coverage of runtime violations from 71.1% to 80.2%.

Industry relevance is direct: the authors are affiliated with KRAFTON, with additional affiliations at KAIST and the Korea National University of Arts, and the paper frames the work as supporting scaling of delegated implementation while preserving intended design.

Future Directions

• Extending beyond the 2D Phaser setting at scale. The paper states that Section 4.8 demonstrates the framework's extension to 3D games in Three.js; a natural next step is broadening that extension across the full benchmark and reporting comparative results, which the truncated content does not provide.
• Cutting the remaining unverified share. The pipeline explicitly separates judgment coverage from pass/fail and leaves rules unverified when the required situation is not established, so increasing conclusive runtime evidence for hard-to-reach requirements remains open.
• Weighting and aggregation choices. The paper reports an analysis of the dependency weighting in Appendix F.1.2 and examines alternative axis weights in Appendix F.5 while retaining the default dependency weighting, leaving the question of how sensitive conclusions are to those choices as continuing work.
• Evaluator-model dependence and cost. Evaluator-model comparisons and cost analyses are reported in Appendix G.2 and further analysis of scenario-based replay assessment in Appendix F.2; how much benchmark outcomes depend on the evaluator model and evaluation cost is an open line of inquiry.

Target Audience

This paper is most useful to researchers building benchmarks for agentic software generation and evaluating specification-following; to game industry engineers and technical directors considering delegating implementation from GDDs to coding agents; and to practitioners designing automated QA, replay, and playtesting pipelines for generated games. Readers need enough background to follow contract graphs, multi-axis scoring, and simulated-agent evaluation, so it is best suited to an advanced audience, though the framing that plausible output is not the same as faithful implementation is accessible to a general technical reader.

Authors’ abstract

Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce A2Z GameSpec-Bench, a benchmark of 100 long-form GDDs for evaluating end-to-end game development by agents. We measure faithfulness by checking whether the game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a dependency-aware contract that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated test policies for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure detection. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond implementation judgments and provides targeted feedback to support more faithful game development. Code and datasets are available at https://a2z-gamespec-bench.github.io.