Wizard: Manipulate Any World in Real Time via Agentic Control

AI summary

Overview

• Research area: Computer Vision — video world models, agentic control, and interactive game generation.
• Technical level: Advanced (assumes familiarity with diffusion/flow-based video generators, autoregressive video generation, distillation, and 3D scene/character representations).
• Scope: The paper introduces Wizard, an agentic game world model that combines articulated character skeletons and scene geometry proxies with a real-time video generator to support fine-grained, zero-shot-controllable gameplay at 24 FPS.

What This Paper Is About

Existing game world models either restrict players to fixed action sets or depend on motion patterns seen during training, so they generalize poorly to new scenes, characters, and actions, and they offer limited real-time interactivity. The authors build a system where an "agent" explicitly maintains the game state — scene geometry, character skeletons, and combat state — and passes that state to a video generator as structured visual proxies (rather than raw pixels or text alone). The goal is a playable world model where a player can supply a scene image and character references and get precise, mechanically consistent, real-time control over motion and the environment.

Key Contributions

1. Wizard, an agentic game world model that enables real-time, zero-shot generalization across unseen scenes, characters, and motions with fine-grained control. The authors describe it as the first agentic game world model to control characters via articulated skeletons and scenes via geometry maps simultaneously.
2. A diverse dataset of 50.3 hours of aligned proxy–video pairs spanning 750 scenes and 178 characters, drawn from public videos, game footage, synthetic rendering, and Unreal Engine simulations, all converted into a unified representation of scene geometry plus articulated character skeletons.
3. An agentic video-to-video framework that maintains explicit world state (character poses, health, distance, orientation, cooldowns) and translates natural-language and camera inputs into updates to structured proxies, enforcing gameplay rules consistently.
4. Character-aware adaptive attention masking, which uses skeleton positions available at both training and inference time to restrict each reference image's attention to the corresponding character region (expanded by 25%), improving identity preservation and reducing cross-character appearance leakage.

Main Findings

• Real-time generation: The final model produces rollouts at 24 FPS with only 2 denoising steps, using a KV cache that retains 20 recent latent frames plus one persistent sink frame.
• Zero-shot generalization (qualitative): With a single trained model and no additional fine-tuning, Wizard generates combat rollouts for unseen scenes, characters, and motions, with proxy poses reflected in the generated frames.
• Mechanics-consistent combat: The agentic controller resolves death when a character's health reaches zero, ends combat, and displays the outcome; additional rules can be added flexibly.
• Open-ended creative play (qualitative): Demonstrated capabilities include character summoning, character transformation mid-gameplay via a new reference image, scene manipulation (e.g., raising a platform, inverting the world), and combinations of transformation and scene edits in one session.
• Dataset composition: 8.8 hours of Elden Ring footage (six bosses, 11 regions, more than 20 player outfits), 11.6 hours of synthetic data (133 boss and 29 player assets, 842 Poly Haven environment assets, 15–30 props per scene, camera motion in 95% of samples), 3.3 hours of Unreal Engine footage, and 26.6 hours from WildWorld.
• Training scale: 44,075 clips; 30K steps of bidirectional fine-tuning, 10K steps of causal adaptation, and a student EMA checkpoint deployed after 500 distillation steps.
• Quantitative benchmarks: Not reported. Evaluation is presented qualitatively through figures showing reference images, agent-controlled proxies, and generated rollouts.

Methodology in Plain English

The system separates "what happens in the game" from "what it looks like."

• Structured proxies instead of pixels. For every frame, the system renders a special control image: one channel encodes height, one encodes depth (brighter means closer), and one channel encodes which character a pixel belongs to plus which way it faces. Character skeletons are drawn into the same image as rounded bone segments with joint markers, so characters and scene geometry share a consistent camera and occlude each other correctly.
• Turning a picture into a playable character. Given a character image, Qwen3-VL infers body structure (posture, number and arrangement of legs, arms, wings, tails, heads, left/right assignment), SAM2 segments the character, TripoSG reconstructs a mesh, and UniRig plus a medial-axis fitting procedure produce candidate skeletons. The best-matching skeleton is labeled with body parts so limbs can be matched across characters.
• Motion from a library, not from scratch. Motions are retrieved from a reference library (CombatMotion, WildWorld, Elden Ring assets) from source skeletons with compatible body structure, with a VLM judging whether the movement style suits the target's appearance. Motions are retargeted by matching limb chains, scaling to the target's limb lengths, and applying inverse kinematics for ground contact, then filtered for bad contact, penetration, or unnatural limb crossing.
• Scene manipulation. MoGe-2 recovers a 3D point map and camera parameters from a scene image; characters are removed, a background mesh plus height field is built, and an OpenGL offscreen renderer re-renders the geometry from the current viewpoint as the camera moves. A Qwen3-VL-8B-powered agent turns natural-language instructions into programs that edit this geometry.
• Combat logic. A shared world state tracks positions, poses, health, action phases, and cooldowns. A behavior planner picks motions — approach when out of range, otherwise simulate candidate swings against the player and favor likely hits within cooldown/phase constraints — while a rule-based controller checks swept hitboxes, respects invulnerability windows, applies damage, and ends combat at zero health.
• Video generation. The generator builds on Wan2.2-TI2V-5B, with the first frame as a clean latent prefix, proxy latents concatenated channel-wise with noisy latents, and character references appended as extra tokens. Skeleton-derived masks gate reference attention per character per frame. Real-time inference comes from a three-stage pipeline following Causal Forcing++: teacher-forced causal adaptation, consistency distillation with an EMA target, then distribution matching distillation on the student's own rollouts.
• Initialization cost: Under five minutes per session to reconstruct the scene and characters, rig skeletons, and build the target-specific motion set.

Why This Matters

Impact on research. Wizard shows a concrete way to decouple game logic from rendering in a video world model: because state lives in an explicit agent rather than in generated pixels, the system gets persistent character identity, rule-consistent combat, and controllable scene edits without retraining. It extends agentic control from simple locomotion and driving to articulated character animation, and it shows that causal adaptation plus few-step distillation can make proxy-conditioned generation interactive.

Real-world applications:

• Player-authored games and prototyping: users sketch a scene and character and immediately play a controllable scenario without an engine pipeline.
• Rapid game design iteration: designers can test layouts, boss behaviors, and movesets in generated footage before committing to asset production.
• Interactive storytelling and animation previews: character transformation and scene manipulation let creators compose narrative beats visually.
• Simulation and training scenarios: the state-tracking combat controller could inform configurable agent-vs-agent environments.

Industry relevance. The work targets the cost of authoring custom game content — specifically, letting a small amount of user input (one scene image, a character reference, a text instruction) drive a generalizable renderer. Tencent's involvement and the emphasis on real-time FPS and latency point directly at interactive entertainment use cases.

Future Directions

• Motion coverage for unusual bodies. The authors note that reliance on a motion library makes it hard to find compatible motions for characters with uncommon body structures; more flexible motion generation is an open problem.
• Long-rollout scene consistency. The paper reports that consistency between scene proxies and generated video may decline over longer rollouts, motivating work on maintaining stable scene appearance over extended play.
• Richer, more realistic motion. Future work aims at character motions that better reflect actual gameplay, moving toward fully playable closed-loop worlds.
• Evaluation beyond qualitative figures. No quantitative benchmarks or metrics are reported here, so standardized measures of control fidelity, identity preservation, and temporal consistency remain to be established.

Target Audience

Researchers and practitioners in generative video, world models, and game AI who already understand diffusion-based video generation and 3D character representations; also useful for game engineers and technical directors evaluating agentic, generation-based alternatives to engine rendering, and for students looking for a clear example of decoupling explicit state from neural rendering.

Authors’ abstract

Recent advances in video world models have enabled the generation of increasingly realistic and interactive virtual environments, opening new possibilities for player-customizable gameplay. Agentic frameworks further advance this direction by decoupling explicit game logic and controllable world representations from visual generation. However, existing approaches support only simple game settings and predefined motion patterns, generalize poorly to unseen environments and characters, and offer limited real-time interactivity. In this paper, we introduce Wizard, an agentic game world model that enables real-time, precise control over character motions while generalizing zero-shot across novel scenes, characters, and actions. To achieve this, we construct a diverse dataset spanning 750 scenes and 178 characters, incorporating skeleton and geometry information as structured representations. We leverage these representations as intermediate proxies to condition a video generation model, together with an adaptive masking mechanism that improves alignment between the conditioning signals and generated video rollouts. Furthermore, we introduce an agentic framework that explicitly maintains and updates game states, allowing players to freely configure their virtual worlds and exercise fine-grained control over character motions while preserving visual quality and gameplay consistency. By combining agentic control with structured proxy-based generation, our framework unlocks new possibilities for player-driven game creation with real-time, generalizable, and mechanically consistent world simulation.