ActionRoPE: action as coordinates for spatially consistent control in planar-camera game world models

AI summary

Overview

Research area: Computer Vision — generative video world models, action-conditioned video diffusion, and geometry-aware positional encoding.

Technical level: Intermediate. The paper assumes familiarity with diffusion/flow-matching video transformers, rotary positional embeddings (RoPE), and latent-space tokenization, but its core idea (actions as world coordinates) is explained geometrically.

Scope: The paper proposes ActionRoPE, a parameter-free interface that converts action-driven planar camera displacement into world-coordinate rotary positions for video tokens, plus a coordinate-conditioned noise schedule, and evaluates both on a top-down 2.5D game dataset across four backbones from 1.3B to 33B parameters.

What This Paper Is About

Existing game world models inject player actions as features — tokens, additive embeddings, adaptive normalization, or cross-attention — which forces the model to learn from data how far and in which direction each action should move the visual world. In "planar-camera" worlds such as top-down and isometric games, that motion is not something to be learned: once the action-to-camera scale is calibrated, an action maps to a geometrically fixed translation of the scene in the image plane. The paper's goal is to replace learned feature conditioning with explicit coordinate alignment, so that the same world location keeps the same positional address across frames, and to exploit the resulting known-versus-newly-revealed region structure to improve denoising.

Key Contributions

1. ActionRoPE — action as world coordinates. A parameter-free action interface that adds the cumulative camera displacement to each video token's spatial RoPE position, so tokens for the same world location share the same address across time. It turns action control from learned feature conditioning into explicit coordinate alignment and adds no parameters to the control path.

2. Coordinate-Conditioned Noise Scheduling (CNS). Using the world coordinates to define a binary "knownness" field (tokens inside the first frame's world footprint versus newly revealed regions), CNS assigns token-wise denoising schedules so observed regions stabilize earlier while new content is generated with reference to them. It requires only a 0.015M-parameter input channel and no extra inference passes.

3. An evaluation protocol for action control covering motion fidelity (slope and R² of measured versus commanded displacement), directional compliance (median angular error), and closed-loop revisit consistency (PSNR, SSIM, NCC, DINOv2 similarity, and residual geometric drift in pixels). The paper reports that its method outperforms four representative action-conditioning mechanisms on the same backbone and reduces revisit drift by 66% over the strongest baseline.

4. Evidence of generality. The same interface is applied to backbones ranging from 1.3B to 33B parameters and to out-of-distribution scenes, supporting the claim that control comes from the coordinate interface rather than a specific architecture or training visual domain.

Main Findings

• Precise motion tracking on the main backbone. On Wan2.2-TI2V-5B, the full model reaches a commanded-to-measured motion slope of 0.981 with R² = 0.952, versus a best baseline slope of 0.911 (Wan + Prompt), and a median directional error of 0.68 degrees.

• Speed scaling is nearly exact. Across commanded speeds of 0.5×, 0.75×, 1.0×, 1.25× and 1.5× the nominal speed, the median measured-to-commanded displacement ratio stays between 0.96 and 0.99. The paper reports that baselines overshoot slow commands and undershoot fast ones, and that Prompt and Cross-Attn move by nearly the same amount at every speed because they receive only direction information (which explains their negative R² in the main table).

• Best revisit consistency. After a 2.5-second walk away and a return along the same path, the method obtains PSNR 23.94 dB, SSIM 0.691, NCC 0.831, DINOv2 similarity 0.892 and residual drift of 7.8 px. Relative to the strongest baseline, the paper reports gains of 2.8 dB PSNR, 13% SSIM and 21% NCC, and a 66% reduction in residual drift (the strongest baseline's drift is listed as 22.6 px at 21.11 dB PSNR).

• Control is cheap in parameters. The method adds zero parameters to the control path and only 0.015M parameters for the known/new mask channel, compared with +0.18M for Linear, +66.5M for AdaLN and +98.6M for Cross-Attn (Prompt adds none but is far less accurate).

• CNS matters, and it matters through training. Removing CNS while keeping ActionRoPE degrades every revisit metric: PSNR 22.81 versus 23.94 dB, SSIM 0.652 versus 0.691, NCC 0.788 versus 0.831, DINOv2 0.885 versus 0.892, and drift rises from 7.8 to 11.7 px. Crossing training and sampling schedules shows the effect is not from the sampler alone: sampling a non-CNS-trained model with the CNS schedule gives 22.82 versus 22.81 dB, while sampling the CNS-trained model with the uniform schedule keeps 24.

Authors’ abstract

Game world models have made rapid progress toward real-time interactive generation. Yet action control is still predominantly implemented as feature conditioning, leaving the model to learn from training data how each action should move the visual world. This formulation is poorly matched to a broad class of planar-camera worlds, such as top-down and isometric games. In these worlds, player motion induces a geometrically well-defined planar camera displacement. We observe that the key to spatially consistent control in such worlds is to represent the effect of each action directly in world coordinates. To this end, we introduce ActionRoPE, a parameter-free action interface that converts action-driven camera displacement into world-coordinate rotary positions. As a result, tokens corresponding to the same world location retain the same positional address across time. These world coordinates further expose a spatial asymmetry in generation: regions covered by the initial observation are geometrically known, while newly revealed regions remain to be synthesized. We therefore propose Coordinate-Conditioned Noise Scheduling (CNS), which assigns token-wise denoising schedules according to this known-new structure. It allows observed regions to stabilize earlier, while new content is generated with reference to them. Compared with four representative action-conditioning mechanisms, our approach follows the commanded motion more faithfully and reduces the revisit drift by 66% over the strongest baseline. The same interface transfers across video backbones from 1.3B to 33B parameters and to out-of-distribution scenes. Project Page: https://teawhite.cn/ActionRoPE/