Code2Games: Enabling Coding Agents for Gaming World Generation

AI summary

Overview

• Research area: Agentic code generation for 3D gaming worlds, spanning computer vision, procedural content creation, and LLM-based software agents (listed as cs.CV, arXiv:2610.05033v1, 04 Oct 2026).
• Technical level: Advanced. The work combines multi-agent orchestration, scene-graph reasoning, vision-language model ranking, and full reconstruction of a game project inside Unreal Engine 5.
• Scope: Code2Games is an agentic framework that turns a natural-language game intent plus a matching base Blender scene into a structured gaming world, then adapts and repairs that world as a runnable Unreal Engine 5 project, evaluated on the new ten-prompt GameCode4D benchmark.

What This Paper Is About

Generating a playable game from a text idea requires keeping scene layout, object placement, gameplay objectives, interactive entities, and executable rules consistent with one another. Existing coding agents can produce individual assets, scenes, or scripts, but the paper argues these pieces are generated and evaluated independently, so a visually plausible object may sit in the wrong region, a gameplay role may bind to the wrong entity, or an objective may sit in an unreachable part of the map. The goal of Code2Games is to keep the game intent, the generated world, and the player's expected interactions tied together through a shared representation with persistent element identifiers.

Key Contributions

1. Code2Games framework. An agentic framework that coordinates scene analysis, gameplay planning, gaming-world generation, and gaming-engine customization to transform a natural-language game intent and a base Blender scene into an interactive gaming world. Work is carried out by four specialized agents: Scene Analysis Agent, Gameplay Planning Agent, Gaming World Generation Agent, and Gaming Engine Customization Agent.
2. Shared scene–gameplay representation with persistent correspondence. Every gameplay element gets a persistent identifier and a correspondence record linking its Blender realization, its Unreal realization, the interactions it participates in, and the executable logic units that implement those interactions, so placement changes, asset swaps, and logic repairs propagate locally instead of forcing regeneration of the whole project.
3. Execution-guided reconstruction. An iterative process that compiles and runs the adapted world, collects engine diagnostics and gameplay test outcomes, localizes the affected components via the correspondence records, and repairs only those components until objectives are met or the repair budget is exhausted. Unreal Engine 5 is the target engine and validation backend.
4. GameCode4D benchmark. A benchmark of ten frozen game prompts specifying genre, viewpoint, and high-level objective, covering varied combinations of scene-structural and gameplay-logic complexity, plus an evaluation protocol for scene consistency, objective grounding, interaction correctness, intent alignment, and runtime executability.

Main Findings

• Code2Games beats matched coding-agent baselines on visual quality and interactive fidelity. Across matched backbones, it consistently improves world quality while maintaining high build reliability. The best configuration, Code2Games with Claude Opus 5, reaches a build reliability of 97.7%.
• Full benchmark table (Table 1) top panel spans five baselines and three Code2Games configurations. The strongest baseline values include Codex + GPT-5.6 Sol at IQ 63.3, Dynamic 83.2, Motion 94.9, Setting 80.7, Inter. 67.8, Physics 69.2, and Claude Code + Claude Opus 5 at Motion 95.1. Code2Games + Claude Opus 5 scores IQ 66.1, Dynamic 88.2, Motion 96.1, Setting 87.5, Inter. 75.8, Physics 73.9. Code2Games + GPT-5.6 Sol scores IQ 65.4, Dynamic 87.2, Motion 95.8, Setting 86.3, Inter. 76.2, Physics 74.4. Code2Games + Qwen3.8-Max scores IQ 64.3, Dynamic 84.8, Motion 94.9, Setting 83.1, Inter. 70.6, Physics 70.3.
• Playable-game quality improves most in content-oriented metrics. Code2Games + Claude Opus 5 records Code 98.1, Image 30.3, Video 34.9, Build 97.7, M 68.8, D 56.7, V 61.0, A 57.0, and Human 77.5. The largest gains relative to the strongest baseline in each column are art and presentation (+15.5), functional visuals (+15.0), content depth (+13.8), and Human (+13.3), versus +9.6 for image quality and +1.0 for motion. The paper reads this as evidence that the main benefit lies in scene-grounded gameplay rather than visual polish alone.
• Metric leadership is split. The Claude Opus 5 configuration achieves the best results on 12 of the 15 evaluated metrics, while the GPT-5.6 Sol configuration leads interaction adherence, physics compliance, and code quality. Even the Qwen3.8-Max configuration exceeds the strongest baseline on 12 of 15 measures and trails only in motion, code quality, and build reliability.
• All projects were repaired successfully. In the experiments, all projects were successfully repaired within five iterations at most.
• Removing scene-grounding components degrades results (Table 2). From full Code2Games (Setting 87.5, Inter. 75.8, Physics 73.9, M 68.8, D 56.7, V 61.0), dropping scene representation mainly reduces Setting (87.5 to 78.2); dropping gameplay specification causes the largest drop in Inter. (75.8 to 69.8); dropping scene constraints most affects Physics (73.9 to 66.2).
• Execution feedback drives build reliability (Table 3). Starting from an Initial reconstruction (Build 78.5, M 64.0, D 53.0, V 58.0, Inter. 58.0, Physics 54.0), adding engine-feedback repair lifts Build to 88.7; adding gameplay-feedback repair lifts Build to 97.7 and Inter. to 75.8. Engine-feedback repair mainly resolves engine-level failures, while gameplay-feedback repair improves interactive behaviour correctness.
• Scene constraints matter progressively (Table 4). Unconstrained realization scores M 58.0, D 45.0, V 52.0, A 45.0; adding region compatibility gives M 62.0, D 50.0, V 55.5, A 50.0; adding geometric feasibility checks gives M 65.0, D 53.0, V 58.0, A 53.0; full scene-constrained selection gives M 68.8, D 56.7, V 61.0, A 57.0. The paper concludes semantic compatibility and geometric validity address complementary failure modes.
• Benchmark composition. GameCode4D spans scenarios including reconnaissance, combat, monster hunting, relic running, racing, skiing, underwater exploration, flight, and resource collection. The complexity figure plots scene entity count (horizontal) against executable behavior count (vertical) on logarithmic scales, with bubble size denoting gameplay element count and color denoting game category (Combat, Exploration, Traversal, Racing), covering named scenarios such as Temple Run, Skiing, Wingsuit, Racing, TPS, FPS, Underwater, Assassin's Creed, Monster Hunt, and Minecraft.
• Evaluation protocol details. For each generated game, the study records two unedited rollouts of approximately 50 seconds, yielding 20 videos per method; the two rollouts execute distinct action routes on the same artifact. Metrics use VBench implementations for image quality (IQ), dynamic degree (Dynamic), and motion smoothness (Motion); WBench for setting adherence (Setting), interaction adherence (Inter.), and physics compliance (Physics); V-GameGym for code, image, and video scores; and GameCraft-Bench for build reliability (Build), core mechanics (M), content depth (D), functional visuals (V), and art and presentation (A). The Human score comes from a user study on a 0–100 scale. Build measures autonomous build-and-run reliability as the average pass rate of standardized engine checks completed without human intervention, compilation failure, or runtime crash.
• Stated limitation. Code2Games encounters a visual fidelity gap when adapting Blender scenes to a game engine: differences in material representations, procedural shader nodes, lighting, and rendering may alter color, surface detail, transparency, and atmosphere, so the Unreal reconstruction may not fully preserve the source Blender appearance.

Methodology in Plain English

The system starts from two inputs: a natural-language game intent and a base Blender scene that was generated from that same intent by Code2Worlds. Rather than asking a coding model to write a game directly, Code2Games builds a shared intermediate representation and then walks it through four agent stages.

First, the Scene Analysis Agent reads the Blender scene and describes it as a set of regions, each with a semantic category, a spatial boundary, an accessibility descriptor derived from the navigation mesh, and a set of interaction affordances. This tells the system where things can go and what can be done there.

Second, the Gameplay Planning Agent converts the intent into a declarative specification of the objective (with success and failure conditions), the actors, the required gameplay elements, and the interactions. Each interaction records who participates, what event triggers it, what precondition must hold, and what game-state update results.

Third, the Gaming World Generation Agent places each gameplay element. Candidate locations are sampled from region boundaries, then filtered by placement constraints and geometric checks: surface support via downward ray casting, collision via object bounding boxes, and reachability via the navigation mesh. A vision-language model then ranks the surviving candidates on a top-down observation to pick the semantically best spot. Physical elements get assets, scale, orientation, and collision geometry; logical gameplay regions get annotations instead of visible assets. Interactions are compiled into executable logic units bound to elements through their persistent identifiers.

Fourth, the Gaming Engine Customization Agent rebuilds the world as an Unreal Engine 5 project: actors and gameplay volumes, materials, lights, environment settings, and runtime conditions and actions for each logic unit. The project is then compiled and run, and execution feedback (compiler diagnostics, runtime errors, and gameplay test failures) is localized to specific components using the correspondence records. Only those components are repaired, and the loop repeats until the objectives pass or the repair budget runs out.

Implementation uses Qwen-family models for language and visual reasoning, Hunyuan3D for gameplay assets, Blender 4.2.0 through its Python API for scene understanding, placement, and visual inspection, and Unreal Engine 5 for the final playable build.

Why This Matters

Impact on research. The paper reframes game generation as a consistency problem rather than a per-asset quality problem. Its persistent-identifier representation and execution-guided repair loop give a reusable recipe for keeping a declarative design, a 3D scene, and engine code aligned across generation stages. It also supplies GameCode4D, a fixed ten-prompt benchmark plus a four-dimension scoring protocol (visual quality, interactive fidelity, multimodal artifact quality, playable-game quality) so that future systems can be compared under identical prompts.

Real-world applications:

• Rapid prototyping of game levels and playable vertical slices from a short text brief, before committing to manual level design.
• Automated production of training or simulation environments where an intent must map to a navigable, interactive world with reachable objectives.
• Regression testing of agentic code pipelines where generated code must compile, run, and pass functional assertions rather than only look plausible in output text.
• Assisting level designers by proposing scene-grounded placements and interaction logic that respect existing spatial and geometry constraints.

Industry relevance. The workflow directly targets the Blender-to-Unreal pipeline used in real production, including material and lighting translation, actor binding, and trigger-volume construction. The measured gains concentrate in content depth, functional visuals, and art presentation rather than image quality alone, which is the axis game studios care about when judging whether generated content is shippable. The reported build reliability of 97.7% for the strongest configuration, with every project repaired within five iterations, speaks to the practical viability of autonomous build-and-run loops in engine-based toolchains.

Future Directions

• Close the visual fidelity gap. The paper's own limitation section calls for improved material translation and rendering calibration to better preserve colour, surface detail, transparency, and atmosphere when moving from Blender to a game engine.
• Reduce dependence on a pre-generated base scene. The framework is conditioned on a Code2Worlds Blender scene generated from the same intent; how much of this pipeline would survive with a weaker or absent base world is not reported.
• Generalize beyond Unreal Engine 5. Engine adaptation is demonstrated only on Unreal Engine 5 as the target engine and validation backend; whether the persistent correspondence and repair mechanism transfers to other engines is not tested.
• Scale the benchmark and the complexity envelope. GameCode4D uses ten frozen prompts and two rollouts per game; whether these results hold for larger prompt sets, longer play sessions, or the higher-complexity open-world scenarios referenced in the figures is not reported.

Target Audience

Researchers and engineers working on LLM/coding agents, procedural 3D content generation, and game AI; game technology teams evaluating agentic pipelines that must produce compilable, runnable engine projects rather than isolated assets or scripts; and benchmark designers interested in evaluation protocols that jointly score visual quality, interactive fidelity, artifact quality, and playability under fixed prompts. Readers should be comfortable with multi-agent system design, 3D scene representations, and game engine concepts such as actors, trigger volumes, navigation meshes, and build diagnostics.

Authors’ abstract

Generating a high-quality gaming world from a natural-language game intent requires joint reasoning about scene structure, spatial layout, gameplay objectives, interactive entities, and executable gameplay logic. Existing coding agents can generate individual assets, scenes, or scripts, but often struggle to maintain consistency across these components. We propose Code2Games, an agentic framework that builds a structured gaming world upon a base Blender world generated from the same game intent. Code2Games coordinates scene analysis, gameplay planning, constrained gaming-world generation, and gaming-engine customization through a shared scene-gameplay representation with persistent element correspondence. After world generation, Code2Games adapts the generated world to Unreal Engine 5 and employs an execution-guided reconstruction process that uses compilation diagnostics, runtime feedback, and gameplay test results to resolve inconsistencies arising during engine adaptation. To systematically evaluate gaming-world generation, we introduce the GameCode4D benchmark, which comprises ten fixed game prompts spanning different levels of scene and gameplay complexity. We evaluate the generated results across four dimensions: visual quality, interactive fidelity, multimodal artifact quality, and playable-game quality. Experiments demonstrate that, compared with direct gaming-world generation by coding agents and existing baseline methods, Code2Games consistently improves the visual quality and interactive fidelity of generated gaming worlds, as well as the quality of the resulting games after engine adaptation.