RAR: Rate-Adaptive Rollout for Time-Consistent Interactive World Models

AI summary

Overview

Research area: Computer Vision, specifically video-based interactive world models and controlled video generation.

Technical level: Advanced. The paper assumes familiarity with autoregressive transformers, rotary positional embeddings (RoPE), KV caching, diffusion/forcing-style video training pipelines, and optical-flow-based evaluation metrics.

One-sentence scope: The paper introduces a framework (RAR) that lets an interactive video world model roll out at controllable, changeable temporal rates by re-parameterizing temporal RoPE in physical-time units, thereby decoupling rollout speed from runtime generation throughput and trading off temporal resolution against physical horizon on demand.

What This Paper Is About

Interactive world models generate a continuous, user- or camera-controlled video stream, but nearly all existing systems fix a single temporal rate (FPS) during both training and rollout. This "temporal-rate coupling" causes two problems: if runtime generation throughput does not match the trained rate, the world evolves too slowly or too quickly relative to the user's expectation; and with a fixed frame budget, a single rate forces a choice between fine motion detail (high FPS, short horizon) and long physical coverage (low FPS, coarse steps). RAR's goal is to make the temporal rate itself a controllable dimension of rollout, so the model can be trained across rates, generate at unseen intermediate rates, switch rates mid-rollout, and cover longer physical horizons at a fixed sequence length.

Key Contributions

1. A formalization of "temporal-rate coupling" as two linked failure modes: rate–throughput mismatch (the modeled world advances at a relative speed ρ = r / f_rollout, where r is runtime throughput and f_rollout the target rollout rate) and the rate–horizon trade-off (physical horizon T = (N − 1) / f under a fixed context of N frames).

2. Stride-RoPE, a modification of RoPE that reparameterizes the temporal rotary coordinate as p_n = n·s(f), with stride s(f) = f_base / f. This makes the relative rotary phase depend on the physical-time interval f_base·(t_m − t_n) rather than on the sampling rate, so equal physical-time intervals induce equal relative phases across rates. It recovers standard RoPE exactly when s = 1, adds no learnable parameters, and only changes temporal coordinates (rotary frequencies and spatial coordinates are untouched).

3. The Rate-Adaptive Rollout framework, which unifies multi-stride training (synchronously sampling video frames and control signals at a set of training strides) with rate-controlled inference, where stride directly sets the target rollout rate and can vary from one transition to the next using accumulated coordinates p_{n+1} = p_n + s_n.

4. Demonstration across three interactive environments and a pretrained model, covering multi-rate generation, generalization to unseen intermediate rates, online rate switching at continuous rollout without clearing the KV cache, improved rate–horizon trade-off under matched training budgets, and transfer to the pretrained SANA-WM world model via finetuning.

Main Findings

• Multi-rate generation is temporally consistent (Q1). At matched physical times, RAR produces similar scene evolution and action progress across rates, including unseen intermediate rates. Under a fixed rollout budget (fixed number of generated frames), lowering the rollout rate from highest to lowest extends the physical horizon by factors of 5, 4, and 3 on Doom, CS2, and FPV respectively, with LPIPS increases of only 6.1%–11.4% relative to the respective highest-rate rollouts.

• Rate fidelity is high and generalizes to unseen rates. Mean |SR − 1| is 3.5% on Doom, 3.5% on CS2, and 4.0% on FPV. Across all six unseen intermediate rates, SR ranges from 0.960 to 1.024, a maximum deviation of 4.0% from the ideal value of 1, comparable to the training rates.

• Online rate switching tracks the target (Q2). On FPV, fitted responses to a smoothly varying stride schedule and an abrupt six-segment schedule yield Pearson correlations of 0.998 and 0.970 respectively, with the KV cache retained throughout.

• Multi-rate training eases the rate–horizon trade-off (Q3). On Doom, under 10k matched updates with a 213-frame window and a 70 FPS single-rate baseline, RAR improves LPIPS at 14 FPS (0.4376 → 0.3834) and 35 FPS (0.3863 → 0.3413) while leaving 70 FPS nearly unchanged (0.3113 → 0.3135). Under approximately matched wall-clock time, RAR reaches lower LPIPS than longer-window models: 0.3383 vs 0.3500 at 425 frames and 0.3552 vs 0.3745 at 849 frames. The 425-frame and 849-frame training configurations cost 1.56× and 5.25× as long per update as the 213-frame configuration.

• The method transfers to pretrained world models (Q4). After a two-stage adaptation of SANA-WM (3k steps bidirectional teacher on 10/20/30 FPS, then 3k steps multi-rate Self-Forcing), LPIPS does not increase at any evaluated rate (10 FPS: 0.5702 → 0.4956; 16 FPS: 0.5684 → 0.5009; 20 FPS: 0.5707 → 0.4979; 24 FPS: 0.5673 → 0.5022; 30 FPS: 0.5678 → 0.4999). SR improves at 10, 20, and 30 FPS and is effectively unchanged at 24 FPS, with 16 FPS being the baseline's native rate.

• RAR outperforms alternative rate-control schemes. Against Single (base-rate-only training with inference-time Stride-RoPE) and Embed (multi-rate training with frame-index RoPE plus an explicit Fourier FPS embedding), SR MAE is 3.54% (Doom), 3.54% (CS2), and 4.02% (FPV) for RAR, versus 13.92%/7.24%/9.40% for Embed and 109.08%/52.24%/67.28% for Single. Averaged over nine seen and six unseen environment–rate pairs, Embed and RAR obtain seen-rate MAEs of 7.31% and 5.09% but unseen-rate MAEs of 14.50% and 1.62%.

• Perceptual quality metrics hold across rates. DOVER scores remain broadly comparable across rates within each environment, and Drift (last-quarter vs first-quarter LPIPS) is reported per rate setting in the main multi-rate table.

Methodology in Plain English

The paper starts from a simple observation about positional encoding. In a transformer with rotary position embeddings, the interaction between a query at frame n and a key at frame m depends only on the index difference m − n. For video sampled at a fixed rate, that index difference maps to a physical-time interval (m − n)/f — so the same physical-time gap produces a different rotary phase depending on FPS. The fix is to place the rotary coordinate at n·s instead of n, where s = f_base/f. The coordinate then measures elapsed physical time in units of 1/f_base, and the relative phase becomes f_base(t_m − t_n) for any sampling rate. This is the entire change: one substitution in where the temporal RoPE is applied.

Training then simply samples video and its control signals at a chosen stride from a training set (for example, strides corresponding to {14, 35, 70} FPS in Doom), always keeping the same number of frames N. Because the frame count is fixed, a coarse stride covers a longer physical duration while a fine stride covers a shorter one — so a single training run sees both fine-grained local motion and long-horizon evolution without lengthening sequences.

At inference, choosing a stride s_rollout sets the rollout rate f_base/s_rollout. If runtime throughput r falls inside the trained range, setting s_rollout = f_base/r makes the relative progression ρ equal to 1, so the generated world advances at the intended wall-clock pace. Because the coordinate p_n = n·s is continuous and valid for any real s > 0, intermediate rates never explicitly trained can be used. For mid-rollout changes, positions are accumulated (p_{n+1} = p_n + s_n); with the Unified Relative RoPE Recipe, the cache stores pre-RoPE keys and values, and rotary rotations are applied to those cached raw keys at each attention step, so the rate can change without clearing the cache or recomputing history.

The experiments use Doom (data generated with ViZDoom and Sample Factory) and Counter-Strike 2 (built from game demos) as discrete keyboard-and-mouse controlled first-person games, and FPV (public Human-Piloted Drone Racing dataset) as a continuous camera-trajectory-controlled drone racing simulator. Doom and CS2 models are initialized from Matrix-Game 2.0 and FPV from SkyReels, trained through Solaris's three-stage pipeline (bidirectional training, Diffusion Forcing, Self-Forcing). Model sizes are approximately 1.8B parameters each for Doom and CS2 (360p) and 1.3B for FPV (480p). Base rates are 70, 64, and 60 FPS with training rate sets {14, 35, 70}, {16, 32, 64}, and {20, 30, 60}, and sequence lengths 213, 241, and 241 frames respectively. Portability is tested on the 720p, 1.6B-parameter SANA-WM with f_base = 30 and training rates {10, 20, 30}.

Evaluation uses LPIPS (reference-based perceptual quality), DOVER (no-reference overall perceptual quality in [0, 1]), and a new rate-fidelity metric, the Speed Ratio (SR), computed from RAFT optical flow normalized by physical-time intervals and aggregated by median across cases against ground truth. SR = 1 means the generated and ground-truth videos share the same cross-rate temporal scaling.

Why This Matters

Impact on research. The paper reframes temporal rate as a controllable capability rather than a fixed hyperparameter, and shows the fix can be a coordinate substitution in RoPE rather than a new architecture. It introduces SR as a rate-fidelity metric and reports a clean head-to-head against FPS-conditioning-style rate control (Embed), showing that continuous FPS embeddings alone interpolate poorly to unseen rates (14.50% vs 1.62% unseen-rate MAE). The transfer result on SANA-WM suggests the technique is portable to existing pretrained world models rather than requiring from-scratch training.

Real-world applications:

• Game engines and neural simulators that must run at whatever frame rate the host hardware can sustain, keeping in-game motion at the intended physical pace.
• Robotics and drone simulation, where a camera-controlled world model (the FPV setting here) must reproduce physical-time-correct dynamics under varying compute budgets.
• Training environments for reinforcement learning agents, where a single model can serve both fine-grained short-horizon rollouts and coarse long-horizon rollouts.
• Content generation and virtual production, where the same model can trade temporal detail for coverage depending on the target output.

Industry relevance. Deployment on heterogeneous hardware is the central practical constraint: a model that assumes a fixed FPS breaks whenever throughput differs from the training rate. RAR's ability to run at unseen intermediate rates, switch rates mid-rollout, and cover longer horizons under matched training budgets addresses both inference-time adaptation and training-cost concerns, which matters for any organization serving interactive generation across different GPUs or edge devices.

Future Directions

1. Extrapolation beyond the training range. The paper reports interpolation results in the main text and references an extrapolation appendix (Table D) computing SR outside the training range, with per-case adaptation of the SR definition; the truncated content does not include those table values, so how well RAR extrapolates is not fully quantified in the available text.

2. Fidelity at large temporal intervals under limited training budget. The paper notes that residual under-scaling at 10 FPS in the SANA-WM adaptation may reflect the difficulty of adapting to larger temporal intervals with only 3k + 3k steps, and cites optical-flow measurement uncertainty as a possible contributor.

3. Interaction with context-window and positional-extrapolation techniques. The related work distinguishes RAR from RoPE-rescaling and positional-interpolation methods that extend context length, but the paper does not report combining RAR with those approaches.

4. Broader control modalities and environments. The evaluation covers two discrete keyboard-and-mouse games and one continuous camera-trajectory simulator; generalization to other action spaces, other simulators, and open-world settings is not measured in the reported content.

Target Audience

Researchers and engineers working on interactive video generation, neural game engines, and world models — particularly those concerned with real-time deployment under variable compute, positional-encoding design for video transformers, or long-horizon autoregressive video rollout. Readers with a background in transformer positional embeddings and diffusion/forcing-style video training will get the most out of the Stride-RoPE derivation and the multi-rate training formulation; practitioners focused on inference-time serving will find the throughput-matching and KV-cache-preserving rate-switching results most directly applicable.

Authors’ abstract

Video-based interactive world models aim to enable real-time, user-controllable environments. Yet most existing systems are trained and rolled out at a fixed temporal rate, limiting both runtime-rate adaptation and physical-horizon coverage -- a limitation we term temporal-rate coupling. Specifically, when runtime generation throughput differs from the learned temporal rate, the generated world evolves unintentionally more slowly or quickly. Moreover, using a single temporal rate imposes an inherent trade-off between temporal granularity and physical horizon: higher rates capture finer motion but cover a shorter duration under a fixed context window. To address both limitations, we introduce Rate-Adaptive Rollout (RAR), a framework that enables interactive world models to roll out at controllable temporal rates. At its core, Stride-RoPE aligns latent frames sampled at different rates on a shared physical-time coordinate, enabling consistent multi-rate rollout, unseen-rate generalization, and online rate switching without discarding history. Meanwhile, RAR uses multi-stride video sampling to enable joint learning of fine-grained dynamics and long-horizon evolution. Across action- and camera-controlled environments, RAR achieves temporally consistent multi-rate rollout and online rate switching while preserving high-rate generation quality and extending physical horizons under matched training budgets. RAR also transfers to pretrained world models while preserving perceptual quality.