AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark

AI summary

Overview

Research area: Computer vision and generative video, specifically video-to-video generative rendering for games, plus a new evaluation benchmark for entity-level correspondence.

Technical level: Advanced. The method requires familiarity with diffusion/flow-matching video generation, LoRA fine-tuning, diffusion transformers, and detection-and-tracking-based evaluation pipelines.

Scope (one sentence): The paper presents AnyGameRenderer, a single LoRA-tuned 33B video generation model that re-renders crude gameplay footage from many game genres into polished visuals while keeping every entity's count, position, and timing, together with a 59-case, 753-entity benchmark that measures exactly that entity correspondence.

What This Paper Is About

AI coding agents can now build playable games in minutes, but the results look like "programmer art": gray blockouts, colored geometric primitives, and flat semantic blobs, because good art normally takes studios months of modeling, texturing, and lighting. The paper's goal is a rendering layer that takes such crude gameplay footage as plain video and outputs the same gameplay in a prompted visual style, frame-aligned with the input, with characters and effects added but layout, camera, and every entity's position and timing preserved. A second goal is measurement: existing renderers are judged at the frame level, which cannot tell whether each individual entity was rendered correctly, so the authors build an entity-level benchmark.

Key Contributions

1. A video-to-video generative renderer for games. A single checkpoint handles a wide range of genres (tower defense, auto-battlers, top-down survivors, turn-based battles, kart racing, crowd runners, arena survival, co-op games, and shooters) and diverse input types (gray blockouts, colored primitives, semantic maps, wireframes, coarse clay figures), and renders them in diverse prompted styles while preserving entity correspondence.
2. A paired training dataset from three sources totaling 10,330 clips. Multi-stream rendering data from 38 agent-made games exported from GEAR (3,221 clips), two-stream rendering data from Unreal Engine with programmatically controlled crowds, count changes, and entity scales (3,910 clips), and real gameplay recordings reconstructed via 3D-box detection (3,199 clips), amounting to about 14.8 hours of paired video.
3. An entity-level benchmark and metric suite. 59 held-out cases with 753 entities, each annotated with ground-truth boxes, scored with temporal match, position match, unrendered rate, and hallucination rate, plus secondary statistics for on-time appearance, never-appearing entities, and missed count changes.
4. Demonstrated correspondence gains over the zero-shot base model. Temporal match rises from 50.9% to 62.9% and missed count changes fall from 34.7% to 21.3%, with improvements on all four main metrics.

Main Findings

• Overall entity correspondence improves across the board. On all 59 cases, temporal match rises from 50.9% to 62.9%, position match from 57.1% to 61.9%, the unrendered rate falls from 4.1% to 2.9%, and the hallucination rate falls from 52.1% to 46.0%.
• Secondary statistics follow the same trend. On-time appearance rises from 55.9% to 67.6%, entities that never appear fall from 26.0% to 20.2%, and missed count changes fall from 34.7% to 21.3% (out of 75 count changes in the benchmark).
• The synthetic Unreal Engine split is nearly solved. On the 11 two-stream rendering cases, temporal match reaches 96.6% against a GT floor of 98.5%, position match reaches 80.6%, the unrendered rate drops to 1.4% from 4.9%, the hallucination rate drops to 19.4% from 38.9%, and no entity is missed (never-appears falls from 15.2% to 0.0%).
• Gains are largest where the base model struggled most. On GEAR's 34 multi-stream rendering cases, temporal match rises by 13 points (47.6% to 60.7%) and missed count changes nearly halve (40.7% to 23.7%); the hallucination rate there improves only modestly, from 55.4% to 51.2%.
• Real recordings improve the least. On the 14 real gameplay cases, temporal match rises only from 46.5% to 51.1%, the unrendered rate is essentially unchanged (2.6% to 2.9%), and missed count changes stay at 20.0%.
• The measurement pipeline itself has a ceiling. The GT floor, which runs the ground-truth RGB video through the same detection and matching pipeline, scores 80.9% temporal match, 79.3% position match, 0.1% unrendered, 34.1% hallucination, 93.6% on-time appearance, 0.7% never appears, and 0.0% missed count changes, showing that even perfect video is not scored perfectly.
• Three characteristic zero-shot failure modes are identified and fixed. The base model leaves crude shapes unrendered (blue cubes, cyan cylinders, a blue cylinder where an elk should be), drops the input viewpoint for out-of-domain styles (a top-down blockout becomes a frontal anime group shot or a first-person voxel scene), and adds characters named in the caption that have no block in the input.
• Generalization varies by genre. A single checkpoint handles all five benchmark genres (first-person shooter, top-down, third-person, crowd, vehicle); temporal match improves in every genre with the largest gains on crowd and third-person games; the unrendered rate falls in every genre except crowd games, where both models are already near zero; position match improves most on vehicle games; the hallucination rate falls on crowd games but rises slightly on top-down and third-person games; top-down games remain the hardest for both models.
• Deliberate caption noise helps rather than hurts. Captions list every character that may appear in a game even if some never show up, which teaches the model to decide what exists from the video; as a result the model rarely adds caption-named characters that are missing from the input.

Methodology in Plain English

The authors treat game rendering as reference-conditioned video editing. The crude gameplay clip is fed to a video generation model as a reference video, and a text caption says what each crude shape stands for and what style to render in. An optional anchor image (at training time, the target video's first frame) sets the style. The crude clip fixes structure; the caption and anchor set the look, so switching styles only requires changing the prompt.

The base model is the ref2va mode of MiniMax-H3, a 33B reference-video-conditioned video generation model that runs at 24 fps up to 768p. The authors freeze the base weights and train only a rank-128 LoRA on all transformer blocks with a standard flow-matching objective; audio loss is disabled and audio discarded at inference. All results use 20-step Euler sampling on base plus LoRA.

A key design choice is the caption format. Captions keep the base model's six segments (subject_definitions, summary, retention_analysis, detailed_description, overall_soundscape, non_diegetic_music) and follow four rules the authors call a "correspondence contract": each subject is described as the real entity with the crude shape named as its representation; the retention segment declares position, apparent size, timing, and head count as exactly preserved while appearance is left to the prompt, and describes empty space positively; all statements are positive because the base model has no negative prompt and negative phrasing tends to produce what it names; and captions are generated automatically from engine metadata and detection results, with deliberate noise included.

Training data is unified to 124-frame clips (5.17 s at 24 fps) at 1344×768 with a silent 32 kHz audio track. Optimization uses a global batch size of 16, a learning rate warming up over 100 steps then cosine decaying with a peak between 2×10⁻⁵ and 2.8×10⁻⁵, weight decay 0.01, bf16 precision, gradient checkpointing, and block swapping, with FlashAttention-3 speeding each step by about 30%. Training runs on 16 NVIDIA H200 GPUs (141 GB), about 129 s per step, and three epochs over the 10,330 clips take about 48 hours.

The Unreal Engine data is scripted specifically around the base model's failures: same-color groups of 2 to 20 characters that walk in, spawn, walk out, or despawn; characters of different colors entering and leaving; entity sizes from 20 to 600 pixels; and mixed camera views (35% follow, 30% top-down, 20% isometric, 15% side). The protagonist is always blue and other characters use a six-color set.

Evaluation runs the generated video through SAM3 open-vocabulary segmentation prompted with the caption's subject phrases plus generic person and vehicle terms, removes shadow and reflection detections, links detections into tracks, and matches them to ground-truth input boxes with the Hungarian method, accepting a pair only if IoU ≥ 0.3 or the centroid offset is within half the input box size. Entities existing for fewer than 12 frames are ignored. Deployment uses a LoRA multiplier of 1.0 for in-domain inputs and styles, and 0.7 for out-of-domain text-only styles.

Why This Matters

Impact on research. The paper shifts the evaluation target for generative game rendering from frame-level agreement (edges, point tracks, pairwise judges) to entity-level correspondence. The authors point out that frame-level metrics barely change when a small entity is missing or a group arrives late, and that edge agreement can even reward leaving crude blocks unrendered. The benchmark and metric suite give the subfield a way to score the bookkeeping that actually matters in games.

Real-world applications:

• Fast prototyping and "vibe gaming": an agent writes the game logic and the engine emits a blockout, and the renderer produces final-looking visuals without modeling, texturing, or animation.
• Style switching without re-authoring assets: a game can change its entire visual style by changing only the prompt (optionally with a style anchor image), rather than rebuilding art.
• Upgrading existing agent-made games: crude footage from tools like GEAR can be re-rendered into higher-quality visuals while keeping gameplay identical.
• Rendering from cheap intermediate representations: because the input is plain video, pipelines can feed blockouts, semantic maps, wireframes, or 3D-box proxies reconstructed from real footage, avoiding engine G-buffers or per-object class labels.

Industry relevance. The work sits alongside concurrent engine-plus-generative-renderer systems such as CWM, PWM, Matrix-Game 4.0, StreamRender-H3, Marionette, and SolarRenderer. Its differentiators are genre breadth, acceptance of plain video instead of a fixed intermediate representation, and being the one that measures entity-level correspondence. The reported cost (16 H200 GPUs, about 48 hours for three epochs) and the single-LoRA deployment make the approach legible to studios considering a generative rendering layer.

Future Directions

• Closing the real-footage gap. Real recordings improve least (temporal match 46.5% to 51.1%, missed count changes unchanged at 20.0%), so better detection-based reconstruction or real-world training data is an open need.
• Fixing top-down and third-person hallucination. The hallucination rate rises slightly on top-down and third-person games while falling on crowd games, and top-down games remain the hardest for both models, leaving room for targeted data and caption design.
• Raising the measurement ceiling. The GT floor of 80.9% temporal match and 79.3% position match shows the detection-and-matching pipeline itself limits measurable scores; better segmentation and tracking would make the benchmark more discriminative.
• Extending beyond single-clip correspondence. The paper evaluates 124-frame clips (5.17 s); whether entity counts, scripted events, and identity hold over longer horizons and under interactive play is not reported and remains an open question.

Target Audience

Researchers and engineers working on generative video, world models, and neural rendering for games; developers of AI game-creation tools and agentic coding systems who need production-quality visuals from crude engine output; and benchmark designers interested in entity-level evaluation of generative models. Readers need a working understanding of diffusion/flow-matching video models and LoRA fine-tuning to get the most from the method section, while the problem framing, failure analysis, and benchmark design are accessible to game developers and technical artists.

Authors’ abstract

AI coding agents can now build small playable games in minutes, but the results usually look like programmer art: gray blockouts, colored geometric primitives, and flat semantic blobs. We present AnyGameRenderer, a video-to-video generative renderer that turns such footage into high-quality visuals with characters, motion detail, and effects. Throughout, it keeps the layout, camera, and every entity's position and timing intact. Our AnyGameRenderer generalizes across a wide range of genres that agents build, such as tower defense, auto-battlers, top-down survivors, turn-based battles, kart racing, and crowd runners. AnyGameRenderer takes plain video as its game-state input, with no complex representations such as engine G-buffers, per-object class labels, or low-poly proxies. Gray blockouts, colored primitives, semantic maps, and wireframes all work as input. It also preserves entity correspondence: group counts stay correct, and entities appear and disappear on the right frame. We implement AnyGameRenderer by LoRA fine-tuning MiniMax-H3 ref2va, a 33B reference-video-conditioned video generation model. Training uses $\sim$10k paired clips from three sources: (i) multi-stream rendering data from 38 agent-made games exported from GEAR, an AI game-creation tool; (ii) two-stream rendering data from Unreal Engine, with programmatically controlled crowds, count changes, and entity scales; and (iii) real gameplay recordings, with pairs reconstructed via 3D-box detection. To evaluate the entity correspondence, we propose an entity-level benchmark of 59 held-out cases with 753 entities, scored against ground-truth boxes. Compared with the base model, AnyGameRenderer better preserves entity correspondence: temporal match rises from 50.9\% to 62.9\%, missed count changes fall from 34.7\% to 21.3\%, and fewer entities are hallucinated or left unrendered.