CtrlCache: Accelerating Interactive Video World Models with Control-Aware Caching

AI summary

Overview

• Research area: Computer Vision — efficient inference for interactive video world models (diffusion-based, chunk-wise autoregressive video generation).
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), few-step denoising, residual caching, and frequency decomposition of latents.
• Scope (1 sentence): The paper proposes CtrlCache, a training-free caching framework that uses the incoming user control sequence to decide when to reuse cached DiT computation and what historical latent content to carry forward, evaluated on Matrix-Game 2.0 and LingBot-World v1/v2 using the WBench navigation track.

What This Paper Is About

Interactive video world models generate video in short chunks so a user can keep steering the simulated world, but every chunk still needs several expensive forward passes through a large diffusion transformer (DiT) backbone. Existing training-free caching methods decide what to reuse from internal model dynamics only, and never look at the user's control inputs. CtrlCache closes that gap by deriving its reuse and refresh schedule directly from controls, which are known before a chunk is denoised and therefore cost no forward pass.

Key Contributions

1. Control-regime latent analysis. The authors measure cosine similarity between clean latents of adjacent chunks under different control signals for three representative sequences, showing that similarity drops around action changes and that the low-frequency component stays more persistent than the high-frequency component during steady interaction.
2. Action-aware scheduling and refresh policy. A training-free mechanism that labels each chunk initial, transition, turning, or steady from the incoming controls, and decides when residual reuse or full computation is used at one selected interior denoising step.
3. Frequency-mixed history prior guidance. A method that builds a prior from the preceding chunk's last clean latent frame, emphasizing its low-frequency structure and attenuating high-frequency detail, then blends it into the current chunk's first-step clean latent estimate during steady chunks only — with no additional DiT forward pass.
4. Evaluation across three world models. Experiments on Matrix-Game 2.0, LingBot-World v1, and LingBot-World v2 report 1.21× to 1.41× DiT-backbone speedups without retraining while improving WBench Overall scores over original inference on all three models.

Main Findings

• Speedups and quality gains across all three backbones. CtrlCache achieves 1.21× to 1.41× DiT-backbone speedups without model retraining and the highest WBench Overall among evaluated methods on all three models.
• Matrix-Game 2.0: CtrlCache raises WBench Overall from 0.6543 (Original) to 0.6786, versus EasyCache at 0.6625 and TeaCache at 0.6495, while delivering 1.05 higher PSNR than the best baseline. Latency is 10.67 s at 1.41× speedup, compared with 14.99 s for Original, 10.33 s for TeaCache (1.45×) and 10.17 s for EasyCache (1.47×).
• Only method improving all five WBench dimensions on Matrix-Game 2.0. CtrlCache reports Quality 0.7370, Cons. 0.6867, Inter. 0.8265, Setting 0.5527, and Physics 0.5900.
• LingBot-World v1: CtrlCache reaches Overall 0.8127 (Original 0.7816) with latency 83.03 s at 1.26× speedup, and the best PSNR (17.27), SSIM (0.5014) and LPIPS (0.3451) among accelerated methods.
• LingBot-World v2: CtrlCache reaches Overall 0.8201 (Original 0.8042) with latency 55.18 s at 1.21× speedup, and the best PSNR (18.35), SSIM (0.5238) and LPIPS (0.2832) among accelerated methods.
• Baselines are marginally faster, but pay in fidelity. TeaCache and EasyCache are slightly faster on every backbone because their reuse criteria can skip computation even at action transitions; CtrlCache instead spends that computation on transition chunks, which is reported as the source of its highest setting adherence and physics compliance on all three backbones.
• Naive residual reuse alone is uneven. Ablating on Matrix-Game 2.0 and LingBot-World v1, plain residual reuse slightly improves Overall (0.6554 and 0.8021 versus 0.6543 and 0.7816 for Original) but degrades physics compliance on Matrix-Game 2.0 relative to Original.
• The action-aware policy helps. Adding it improves Overall to 0.6614 on Matrix-Game 2.0 and 0.8055 on LingBot-World v1, and improves setting and physics compliance on Matrix-Game 2.0 and setting compliance on LingBot-World v1.
• History prior guidance works best restricted to steady chunks. Applying it on steady chunks only gives the best Overall (0.6786 on Matrix-Game 2.0, 0.8127 on LingBot-World v1); extending it to turning (0.6641 / 0.7946) or to steady, turning and transition chunks (0.6559 / 0.7891) reduces performance.
• Qualitative results. TeaCache and EasyCache show distortions in local appearance and scene structure, with an example being the disappearance of an arched doorway in LingBot-World v1, whereas CtrlCache better preserves local details and scene structure.

Methodology in Plain English

The setup. Each video chunk is denoised over a small number of steps, and every step normally runs the whole DiT. During the denoising loop, the model produces a "residual" — the difference between the feature tensor going into the transformer blocks and the one coming out. If a step is skipped, that cached residual is added to the current input instead, while the input projection, timestep embedding, and output head are still computed. The cache is refreshed only when a step is fully computed, and stays fixed across consecutive reuse steps.

Deciding when to reuse (action-aware scheduling). Because controls arrive before the chunk is denoised, they can be inspected for free. The policy compares each control input with the previous one to detect a switch, and separately checks whether a turning magnitude exceeds a backbone-specific threshold. This yields four chunk states: initial (first chunk), transition (a control change occurs, checked both at the chunk boundary and within the chunk), turning (no change, but turning is active), and steady (unchanged, non-turning controls). At one selected interior denoising step, initial and transition chunks run the full DiT computation (initializing or refreshing the cache), while turning and steady chunks reuse the cached residual. The first and final denoising steps are always fully computed.

Deciding what to carry forward (frequency-mixed history prior). The preceding chunk's last clean latent frame is split into a low-frequency part, obtained with 3×3 average pooling (unit stride, replicate padding, applied per frame and channel), and a high-frequency part defined as the remainder. A history prior for each of the chunk's temporal positions combines the two with a high-frequency weight α (α = 1 at the first position, α elsewhere) and a temporal weight w that decays with distance from the preceding chunk. This prior is blended into the clean latent estimate at denoising step s = 1 using a guidance strength λ, and is enabled only for steady chunks, since during transitions and sustained turning the inherited geometry may be outdated. Because it manipulates the clean estimate rather than running the transformer, it adds no DiT forward cost.

Implementation and evaluation. All experiments use a single NVIDIA A100 GPU with random seed 42, and all methods share the same prompts, control sequences, initial frames and decoding settings. Evaluation is on the navigation track of WBench, reporting Quality, Consistency, Interaction, Setting and Physics (Overall is the unweighted mean of the five), plus PSNR, SSIM and LPIPS against the paired Original, and mean DiT-backbone latency excluding KV-cache commit/update time.

Configuration details. Original denoising schedules are [1000, 908, 713] for Matrix-Game 2.0, [999, 978, 947, 825] for LingBot-World v1, and [999, 967, 908, 768] for LingBot-World v2; default residual-reuse timesteps are 908, 947 and 967 respectively. The turning thresholds are τ_turn = 10⁻⁶ for Matrix-Game 2.0 mouse-control magnitude and τ_turn = 0 for yaw/pitch motion magnitude in both LingBot-World variants. Unless stated otherwise, α = 0.5 and λ = 0.5. Temporal weights are (1, 0.5, 0.25) for Matrix-Game 2.0 and LingBot-World v1 (L = 3), and (1, 0.5, 0.25, 0.125) for LingBot-World v2 (L = 4).

Why This Matters

Research impact. The work reframes training-free diffusion caching as a control-conditioned problem rather than a purely internal-dynamics problem, and shows that a signal available for free (the incoming controls) can guide computation allocation. It also adds evidence that low-frequency latent structure persists across chunks even when high-frequency detail does not.

Real-world applications:

• Interactive game engines and simulated environments where user steering latency directly shapes the experience.
• Robotic or embodied navigation simulators that need responsive, long-horizon visual rollouts.
• Virtual exploration and training scenarios that stream continuously generated video conditioned on user or agent actions.
• Prototyping and content creation tools that let a user drive a generated world in real time.

Industry relevance. The method requires no retraining and no change to model weights, so it can sit on top of already-deployed interactive world models. The reported speedups are on DiT-backbone latency specifically; for LingBot-World v1, acceleration is most valuable since Original latency is 104.54 s per case, falling to 83.03 s with CtrlCache. The limitation that reuse steps and action indicators are backbone-specific means deployment currently needs per-model configuration.

Future Directions

• Replace backbone-specific action indicators with more general control representations, so the policy transfers without per-model tuning.
• Move beyond a fixed reuse step that varies with each backbone's denoising schedule toward adaptive step selection.
• Investigate the latency figures reported here, which exclude KV-cache commit/update time, to characterize full end-to-end interactive latency.
• Examine more fluid continuous control signals and longer rollouts, where the boundary-action mechanism (carrying the previous chunk's last control input) is exercised repeatedly.

Target Audience

Researchers and engineers working on diffusion-based video generation, generative world models, and interactive simulation who need faster inference without retraining. It is also relevant to practitioners deploying streaming video generation systems under tight latency budgets, and to readers interested in how control signals can be used as a scheduling signal rather than only as conditioning input. Readers should already understand diffusion sampling, transformer residual caching, and video quality metrics such as PSNR, SSIM and LPIPS.

Authors’ abstract

Interactive video world models need to generate each video chunk efficiently while responding faithfully to user controls. Many systems use chunk-wise autoregressive generation with few-step denoising, but each chunk still requires several costly denoising iterations. Training-free caching can reduce this cost, yet existing policies make reuse decisions primarily from model-internal denoising dynamics and do not explicitly account for control transitions. Actually, interactive generation explicitly exposes a signal they do not use: the controls for a chunk arrive before it is denoised, so a schedule derived from them costs no forward pass. To this end, we analyze adjacent chunks under different control regimes and find that structural similarity drops around action changes, while low-frequency structure remains more persistent than high-frequency detail. Motivated by these observations, we propose CtrlCache, a training-free control-aware caching framework that adapts computation to the current control sequence. Specifically, the action-aware scheduling and refresh policy detects action changes across and within chunks, and labels each chunk as initial, transition, turning, or steady state. At one selected interior denoising step, initial and transition chunks retain full computation, while turning and steady chunks reuse the transformer residual from the most recent fully computed step in the same chunk. To exploit the persistence of low-frequency structure during steady interaction, we further introduce a frequency-mixed history prior guidance that incorporates complementary information from the preceding clean latent without an additional DiT forward pass. Evaluated on Matrix-Game 2.0 and LingBot-World v1/v2, CtrlCache achieves 1.21x to 1.41x DiT-backbone speedups without model retraining while improving WBench Overall scores over original inference across all three models.