RealtimeWAM: How Fast Can I Run My World Action Model?

AI summary

Overview

• Research area: Robotics — real-time inference optimization for World Action Models (WAMs), a class of robot policies that combine visual dynamics prediction with action generation.
• Technical level: Advanced.
• Scope: The paper presents RealtimeWAM, a training-free framework that overlaps observation processing with prediction, reuses observation tokens under hardware-aware constraints, and adapts denoising refinement to predicted motion, evaluated on two WAM backbones across three simulation benchmarks and five real-world tasks.

What This Paper Is About

World Action Models produce good predictive representations for robot control but are slow: on the paper's hardware, native FastWAM inference takes 214.33 ms and native OpenWAM takes 672.83 ms per request, which is too slow for a robot to react to a scene that changes while the model is still computing. Existing fixes either require a specially tailored architecture (as in FastWAM, which skips future-video generation at test time) or rely on generic caching that does not account for the changing computational needs of closed-loop control. RealtimeWAM's goal is to cut this latency across diverse WAM architectures without any additional training, while keeping task success essentially unchanged.

Key Contributions

1. Dependency-aware parallel execution. A schedule guided by layerwise dependencies that overlaps current-observation processing (Prefill) with action prediction and, when present, joint future-video generation, releasing observation K/V layer by layer so prediction can begin before Prefill finishes.
2. Hardware-aware token reuse. A Prefill policy that selects which observation tokens to refresh using RGB change (required positions) plus feature-change ranking (extra positions), choosing the smallest profiled execution capacity that covers both, and exploiting GPU latency plateaus to refresh additional tokens at little extra cost.
3. Motion-adaptive refinement. A scheme that caches Transformer input-to-output residuals and uses the displacement of a coarse action prediction after two refreshes to decide whether to spend two (2F) or four (4F) full Transformer evaluations within ten Euler updates, favoring replanning for large movements and extra refinement for small adjustments.
4. Broad evaluation. Two WAM backbones (FastWAM, OpenWAM) on RoboTwin 2.0 clean/randomized, LIBERO, and LIBERO-Plus, plus five real-world tasks, with controlled ablations of execution order, token allocation, and refinement budgeting.

Main Findings

• Large speedups with near-parity accuracy in simulation. On RTX 4090, measured mean inference latencies are 24.09 ms for FastWAM (8.90x) and 63.09 ms for OpenWAM (10.67x) relative to native inference. Average success rates are 82.75% and 87.41%, within 0.02 and 0.53 percentage points of native (82.77% and 87.94%).
• Where the gains come from. With fixed evaluation budgets (Table 2), adding CUDA Graph replay, compilation, parallel execution, and token reuse reduces FastWAM request latency at 10 evaluations from 214.33 ms to 29.47 ms, and OpenWAM from 672.83 ms to 217.24 ms. First-forward latency after compilation falls from 36.21 to 17.77 ms (FastWAM) and 60.62 to 37.12 ms (OpenWAM) with parallel execution, then to 17.69 and 31.62 ms with token reuse.
• Gains over compilation alone. Relative to Native with compilation and CUDA Graph replay, speedups are 4.14x on FastWAM and 7.15x on OpenWAM.
• Comparison with caching baselines. With compilation and CUDA Graph replay, TeaCache is the fastest competing baseline at 49.28 ms (FastWAM) and 245.99 ms (OpenWAM), with lower average success rates than RealtimeWAM on both. BAC retains a higher OpenWAM success rate (87.77% versus 87.41%) at 305.08 ms.
• Token reuse and selection matter. On the OpenWAM/RoboTwin ablation, the full method reaches 93.4% success at 63.76 ms request latency; disabling token reuse gives 93.8% at 72.10 ms; selecting the refresh set randomly instead of using RGB/feature guidance drops success to 91.3% at 62.32 ms; skipping capacity filling drops success to 92.1% at 61.78 ms. Filling from 65 to 80 tokens raises first-forward latency from 30.76 to 30.96 ms.
• Motion-adaptive refinement beats fixed and random budgets. The motion gate gives 93.4% success at 63.76 ms, versus 91.1% at 64.77 ms for random 2F/4F selection. Fixed 2F reaches 89.6% at 54.70 ms and Fixed 4F reaches 93.7% at 95.26 ms, so the adaptive policy approaches Fixed 4F accuracy at lower mean latency.
• Real-world improvement. Across five real-world tasks, average success rates rise from 35.6% to 52.8% on FastWAM and from 33.6% to 70.8% on OpenWAM (+17.2 and +37.2 percentage points). Mean inference latency falls from 210.71 to 23.72 ms and from 640.87 to 65.27 ms.
• An empirical latency threshold. On the conveyor tasks, success was substantially lower when measured inference latency exceeded approximately 90 ms; this is marked as a dashed line in the paper's Figure 1.
• Numerical agreement. At matched budgets, mean case MAE between Parallel and Native execution ranges from 0.001245 to 0.002515 across configurations, with maximum coordinate differences of 0.031250 or below.
• Preparation scheduling gains are small but real. At a fixed 80-token budget, VAE/condition overlap gives a paired median saving of 0.778 ms (95% interval [0.742, 0.822]) and early RGB routing adds 0.274 ms ([0.225, 0.324]), with identical selected positions and bitwise-identical actions.

Methodology in Plain English

The paper treats a single WAM request as having two separable cost centers: processing the current observation (Prefill) and the repeated denoising evaluations that produce the action chunk.

First, the researchers reorder the work. Because observation tokens are independent of the evolving prediction under first-frame-causal attention, each layer's observation keys and values can be released as soon as they are projected, letting prediction start while observation processing is still running. Fused operators, preallocated buffers, and CUDA Graph replay reduce overhead.

Second, they shrink the observation work that competes with prediction. Observation tokens are cached as FFN outputs. Positions whose RGB content changed beyond a threshold must be refreshed; the remaining positions are ranked by normalized feature change and the best of them fill out a chosen token budget. The budget comes from offline profiling of the concurrent first stage, which shows ranges of token counts where latency barely changes — so the authors pick the smallest profiled capacity that covers the required plus reserved tokens, spending otherwise free capacity on tokens that changed.

Third, they adapt the number of denoising evaluations. Both the 2F and 4F schedules share two initial Transformer refreshes (at Euler updates 0 and 3). After the second, a coarse action prediction is decoded, and the maximum predicted end-effector displacement compared to the current position decides whether the extra refreshes at updates 6 and 9 are performed (4F) or skipped (2F). All configurations perform ten Euler updates, and the output head is evaluated at every step with the current timestep.

Evaluation covers native inference, VLA-Cache, TeaCache, ToCa, and BAC as training-free caching baselines, with compilation and CUDA Graph replay enabled for the baselines and for RealtimeWAM. Runtime measurements use BF16, batch size one, and a modified 48 GiB RTX 4090. Real-world experiments use a Franka Research 3 robot, a RealSense D435i camera, 50 trials per task per backbone per configuration, 32-action chunks, and a 20 Hz control frequency.

Why This Matters

Impact on research. The paper reframes WAM acceleration as a scheduling and allocation problem rather than purely a compression problem. It shows that latency is not determined by FLOPs alone — GPU scheduling, kernel quantization plateaus, and the precision actually needed for the current motion all matter. The framework is training-free and applies to multiple architectures, which makes it a layer that could sit on top of future WAMs without architectural surgery.

Real-world applications (drawn from the paper's own task suite):

• Conveyor manipulation, where the paper stacks three shuttlecocks, places a carrot on a plate, and moves a tennis ball into a bowl while the belt runs at 1 m/min.
• Catching and redirecting moving objects, demonstrated by intercepting a rolling can on a tabletop with one edge raised by 5 cm.
• Contact-rich placement tasks such as hanging a mug by its handle on a rack.
• Any closed-loop control setting where the scene evolves during inference, which the paper identifies as the core failure mode of slow WAMs.

Industry relevance. The measured ~90 ms conveyor threshold gives practitioners a concrete latency target: above it, task success dropped substantially in these experiments. The framework requires no retraining and no new checkpoints, so deployment teams can apply it to released fine-tuned models. The gains are largest on the heavier backbone (OpenWAM, 10.67x) and on dynamic tasks, which is where the economic pressure for real-time autonomy is greatest.

Future Directions

• Generalizing beyond the two backbones. The evaluation covers FastWAM and OpenWAM; whether the same profiling-based capacities and motion thresholds transfer to other WAM designs, diffusion policies, or flow-matching policies is untested.
• Removing hand-tuned constants. The change threshold delta, the reserve fraction alpha, the capacity set B, and the motion threshold tau are fixed during deployment or set offline. Learning or adapting these online — without training a selector, which the paper explicitly avoids — is an open question.
• Hardware portability. The capacity set B comes from offline profiling of a specific modified 48 GiB RTX 4090; how the plateaus and capacities shift on other GPUs or accelerators is not reported.
• Cache validity across long deployments. The paper notes that episode or conditioning changes invalidate incompatible history, and that changes to weights, conditioning, camera layout, or preprocessing require compatible cache state. Managing invalidation robustly in long-running or frequently reconfigured settings is left open.

Target Audience

Robotics and embodied-AI researchers working on vision-language-action and world-model policies; systems and inference-optimization engineers who deploy perception-and-control models under hard latency budgets; and graduate students who want a concrete example of coupling execution scheduling with adaptive computation. Readers should be comfortable with Transformer inference internals (Prefill, K/V caching, attention), diffusion or flow-matching denoising loops, and GPU profiling concepts such as wave quantization and CUDA Graph replay.

Authors’ abstract

World Action Models (WAMs) combine visual dynamics modeling with action generation, but their high inference latency limits responsive robot control. Recent efforts accelerate inference by removing explicit future-video generation at test time, as in FastWAM, an approach that requires a specially tailored architectural design. More general caching strategies exploit feature redundancy, but redundancy alone does not capture the changing computational demands of closed-loop control. To address these challenges, we present RealtimeWAM, a general, training-free framework that coordinates parallel execution with adaptive computation for low-latency inference across diverse WAM architectures. We exploit layerwise dependencies to overlap observation processing with prediction. However, concurrent branches still compete for GPU resources, limiting the benefit of parallel execution. We therefore adapt computation throughout the pipeline through selective reuse, caching observation features in visually stable regions and reusing Transformer residuals while reserving additional refinement for small predicted adjustments. We evaluate RealtimeWAM on FastWAM and OpenWAM across RoboTwin, LIBERO, and LIBERO-Plus. On an RTX 4090, measured mean inference latencies are 24.09 and 63.09 ms, corresponding to average speedups of 8.90$\times$ and 10.67$\times$. Average success rates are 82.75% and 87.41%, respectively, within 0.02 and 0.53 percentage points of native inference. Across five real-world tasks, RealtimeWAM improves average success rates over native inference by 17.2 and 37.2 percentage points on FastWAM and OpenWAM, respectively.