Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control

AI summary

Overview

Research area: Computer Vision / generative world models — specifically interactive, action-controllable video generation.

Technical level: Advanced. The report assumes familiarity with diffusion/flow-matching transformers, distillation objectives, autoregressive video generation, and serving infrastructure.

Scope: A technical report describing Zing-0.5, a 5B autoregressive world model that combines magnitude-aware keyboard control with online text instructions and runs at real-time speed.

What This Paper Is About

Interactive world models let users move through generated scenes with keyboard or camera controls, while language-conditioned systems let users describe changes in content or behavior. These two kinds of control are usually offered separately, so users cannot steer with keys while also redirecting the scene through text. Zing-0.5 aims to join both controls inside a single continuous generation session, and to do so fast enough (24 FPS) and cheaply enough that repeated exploration is practical.

Key Contributions

1. Unified action and text conditioning. Magnitude-aware keyboard inputs (continuous strengths on discrete directional keys), temporally aligned text instructions, and jointly annotated videos are combined so that navigation and event control are learned within the same sequence.
2. Event-scale supervision for incremental generation. A segment-level teacher trained on connected multi-prompt videos supervises a block-level causal student through distribution-matching distillation, so an event that spans several student blocks still receives coherent supervision.
3. Low-cost real-time interaction. Four-step generation plus context-preserving streaming supports 832 × 480 inference at 24 FPS at an estimated server rental cost of approximately $0.009 per stream-minute.
4. Open release. Model weights, inference code, and the Zing-SGLang serving implementation are released.

Main Findings

• WBench Navigation score: On the 158-case Navigation split of WBench, Zing-0.5 achieves an overall score of 81.0 and a consistency score of 88.5, with 5B parameters and four denoising steps. Videos were generated at 1248 × 704 resolution with four denoising steps per block and saved at 24 frames per second.
• Leaderboard position: In the paper's Table 1 (official leaderboard results as of September 9, 2026, on a 0–100 scale), Zing-0.5's 81.0 overall matches JoyAI-Echo-1.5 (4-step) at 81.0, behind JoyAI-Echo-1.5 (bidirectional) at 81.6, and ahead of HiDream-O1-World at 80.9, Alaya-EVOKE (3-step) at 80.8, and LingBot-World v2 (fast) at 79.4. Its physical plausibility sub-score of 73.8 is the highest in the table shown; its interaction sub-score of 84.2 is below several listed systems.
• Joint control demonstrated qualitatively: A session in Figure 2 shows a text-directed event change (a sled rider told to cheer and open an umbrella) during continued navigation in a snowy landscape, without restarting generation.
• Serving performance: On a server with 8 RTX 5090 GPUs, Zing-0.5 serves 8 independent 832 × 480 streams at a client-visible 24 FPS, with unpaced steady-state measurement reaching 24.63 FPS.
• Lightweight action branch: The action encoder adds 3.68M parameters, approximately 0.074% of the 5B backbone, and its projection is zero-initialized to preserve pretrained behavior at the start of adaptation.
• Observation on combined DMD/DFD training: Jointly assigning a subset of each batch to Data-Forcing Distillation and the rest to standard DMD reduces fluctuations in motion magnitude and mitigates severe visual degradation, though prolonged training produced some high-frequency noise artifacts and reduced color saturation.
• Additional qualitative behaviors: Released sessions show keyboard navigation across a sea of clouds; a text instruction igniting a silver-flowered tree into a shower of golden sparks; first-person hands directed at a melting candy house; passage through a bedroom mirror into a garden; and text-introduced effects such as a rainbow over a city, fiery rain over a battlefield, a character rising into flight, and a flying mount breathing fire.

Methodology in Plain English

Zing-0.5 starts from a pretrained 5B bidirectional video model (Wan2.2-TI2V-5B) and keeps its autoencoder and diffusion Transformer, then adds an action-conditioning branch and extends text conditioning so prompts can change mid-generation.

• Controls as keys plus magnitude. Instead of converting keyboard presses into camera poses (which can drift as generated motion diverges from the pose trajectory), the model conditions directly on keyboard keys. Each key (W/A/S/D for movement, I/J/K/L for view changes) carries a nonnegative continuous strength, so multiple channels can be active at once. These magnitudes represent relative control intensity, not metric displacement. A small causal encoder averages the per-frame controls into each latent-frame window, embeds the magnitudes, and adds the result to the visual tokens.
• Prompts attached to time intervals. Rather than one prompt for a whole video, each prompt covers a temporal interval, and visual tokens in that interval attend only to that prompt through cross-attention. At inference, a prompt update refreshes only the text K/V cache while the visual K/V cache is retained, preserving scene context.
• Four training stages. (1) Bidirectional adaptation adds action conditioning while mixing in action-free T2I, T2V, and I2V data to preserve pretrained generation ability, followed by a phase on 30-second videos. (2) Autoregressive adaptation trains two branches from the same adapted model: a segment-level teacher that denoises a whole prompt interval with bidirectional attention (used only for supervision, not online), and a block-level generator that produces four latent frames at a time. First-frame synthesis is separated from continuation (T2V becomes T2I then I2V), yielding a nominal 1 + 4n latent-frame layout; both branches are further trained on 30-second videos, and the generator's conditioning history is augmented with clean, noisy, and spatially blurred histories. (3) ODE initialization on the teacher's denoising trajectories, followed by local consistency distillation toward an exponential-moving-average target. (4) Distribution matching distillation, where the student rolls out its own outputs, a frozen real scorer and a learned fake scorer compute detached surrogate gradients (following Decoupled DMD, with Data-Forcing Distillation mixed in probabilistically), and the student replays the recorded states in a packed gradient-enabled pass. The final student uses four denoising steps per block.
• Serving. Each GPU hosts a complete replica (DiT, bounded causal KV cache, local TAEHV decoder), avoiding cross-GPU latent transfer. A bounded visual KV cache uses a fixed prefix sink plus a sliding window, with a "pin" on the first latent frame after each prompt switch so a visual reference survives the instruction change. Frames go straight into a non-blocking bounded media ring that favors the live edge under backpressure.

Why This Matters

Impact on research. The paper treats playability as an explicit design target rather than a side effect of generation quality: users should be able to explore, intervene through language, and use the world's response to decide what to do next. Its main technical argument is that event-scale learning and block-level incremental generation operate at different temporal scales, and that a segment-level teacher supervising a block-level student is a way to bridge them. Its other argument is practical: conditioning directly on magnitude-aware keyboard input avoids the drift that can occur when inferred camera poses fall out of step with generated visuals.

Real-world applications:

• Interactive games and prototype "playable" environments where designers sketch scenes and behaviors in text while steering them live.
• Simulation and training environments where an instructor issues natural-language changes to a running scenario without restarting it.
• Creative and previsualization tools, letting users explore a generated world by keyboard and redirect narrative events by prompt.
• Low-cost streaming experiences, since the reported ~$0.009 per stream-minute points to plausibility for many concurrent sessions on commodity GPUs.

Industry relevance. Serving 8 independent streams on 8 RTX 5090 GPUs, with model weights, inference code, and the Zing-SGLang serving stack released, lowers the barrier for developers building interactive generated worlds rather than passive video generation.

Future Directions

• Persistent consequences. The authors note their examples show visible responses within selected intervals but do not establish whether those consequences survive later interactions or viewpoint changes. An object moved aside should remain displaced when the user returns.
• Explicit world state. Zing-0.5 generates each continuation from visual context, text, and directional actions, and does not separately track entity states or enforce rules governing their transitions; needed information must remain implicit in those representations or be recoverable from them.
• Robustness of the distillation recipe. The paper reports that prolonged joint DMD/DFD training produces high-frequency noise artifacts and reduced color saturation, which invites further work on the objective schedule.
• Extending the joint-control evaluation. The navigation numbers come from WBench Navigation, while the text-directed event control is supported by qualitative sessions; a quantitative protocol for joint action-and-text control would be a natural next step.

Target Audience

Researchers and engineers working on interactive video world models, controllable video generation, and few-step diffusion distillation, as well as practitioners building real-time generative serving systems. The opening framing is accessible, but the training and distillation sections assume comfort with flow matching, diffusion distillation objectives, and autoregressive video generation. Product and technical leaders evaluating deployment cost may also find the inference and servicing numbers relevant.

Authors’ abstract

We introduce Zing-0.5, a 5B autoregressive world model designed for playability: users can explore generated worlds, influence unfolding events, and respond to the resulting feedback through joint keyboard and online text control. Our approach brings together three technical contributions: (1) Unified action and text conditioning, combining magnitude-aware keyboard inputs with temporally aligned text instructions and jointly annotated videos to learn navigation and event control within the same sequence; (2) Event-scale supervision for incremental generation, using a segment-level teacher trained on connected multi-prompt videos to supervise a block-level causal student through distribution-matching distillation; and (3) Low-cost real-time interaction, combining four-step generation with context-preserving streaming to support 832 x 480 inference at 24 FPS at an estimated server rental cost of approximately USD 0.009 per stream-minute. Zing-0.5 achieves an overall score of 81.0 and a consistency score of 88.5 across 158 WBench Navigation cases. A joint-control demonstration shows a text-directed event change during continued navigation without restarting generation. We release the model weights, inference code, and Zing-SGLang serving implementation to support further work on playable generated worlds.