Envisioning the Future, One Step at a Time

AI summary

Overview

Research area: Computer Vision — future scene dynamics prediction, generative world models, and trajectory forecasting.

Technical level: Intermediate. The paper assumes familiarity with transformers, diffusion/flow matching, and autoregressive decoding, though its central idea (predicting sparse points instead of pixels) is conceptually simple.

Scope: The paper proposes an autoregressive diffusion model that predicts open-set future motion as sparse point trajectories from a single image, plus a new benchmark (OWM) and a billiard planning evaluation.

What This Paper Is About

Most world models predict the future by generating dense video frames or dense latent representations, spending most of their capacity on how the scene looks rather than how it moves. This makes it expensive to explore many possible futures, which is exactly what is needed when the world is uncertain and multi-modal. The paper instead formulates future prediction as step-by-step inference over a sparse set of user-specified point trajectories from one image, so that thousands of diverse futures can be rolled out quickly while remaining physically plausible.

Key Contributions

1. Sparse, open-set, step-wise formulation. Visual motion prediction is cast as modeling a distribution over sparse point trajectories (K visible query points in 2D, x_t ∈ R²) from a single image I_0, avoiding any rendering of appearance.
2. An autoregressive diffusion model for this formulation. The model factorizes the joint trajectory distribution causally over time and, within each step, over trajectories, with an efficiency-optimized "Fast Reasoning Blocks" architecture enabling large-scale sampling of diverse futures.
3. The OWM benchmark. A set of 95 diverse in-the-wild videos for evaluating the accuracy and variability of predicted trajectory distributions under open-set, static-camera conditions.
4. Accuracy plus efficiency. The method matches or surpasses dense video simulators in predictive accuracy while being orders of magnitude faster, enabling exploration of thousands of plausible futures within the same compute budget — and it is also applied to zero-shot billiard-shot planning.

Main Findings

• Open-world accuracy beats dense video models. On OWM under the Best-of-5 setting, Myriad (665M parameters) achieves a minADE of 0.029, versus 0.037 for MAGI-1 (4.5B), 0.039 for Wan2.2 (14B), 0.051 for CogVideo-X 1.5 (5B), 0.058 for SkyReels V2 (1.3B), and 0.054 for SVD 1.1 (1.5B).
• The gap widens under a time budget. In the primary Best-within-5-min setting on OWM, Myriad reports 0.013, compared with 0.066 (MAGI-1), 0.068 (SkyReels V2), and 0.119 (SVD 1.1); Wan2.2 and CogVideo-X 1.5 are marked DNF.
• Constrained physical diagnostics. On the PhysicsIQ "solid mechanics" subset, Myriad reports 0.115 (Best-5) and 0.045 (Best-5min); on Physion, 0.048 (Best-5) and 0.020 (Best-5min). MAGI-1 reports 0.126/0.169 on PhysicsIQ and 0.061/0.081 on Physion.
• Throughput advantage. Myriad is listed at 2200 samples/min, against 0.303 (MAGI-1), 0.304 (SkyReels V2), 0.714 (SVD 1.1), 0.141 (Wan2.2), and 0.051 (CogVideo-X 1.5).
• 3D-track training variant. A Myriad variant trained on 3D tracks projected to the first camera view (on roughly 1.5M clips) reports 0.036 (OWM Best-5), 0.020 (OWM Best-5min), 0.117 (PhysicsIQ Best-5), 0.043 (PhysicsIQ Best-5min), 0.048 (Physion Best-5), and 0.028 (Physion Best-5min).
• Billiard planning works best with both sparsity and step-wise unrolling. Under a fixed compute budget, Myriad reaches 78% accuracy at 496.4 actions/min, compared with 8% for full trajectory diffusion (160.8), 4% for the Flow Poke Transformer (13,422.6), 8% for autoregressive images-to-video diffusion (18.6), 16% for images-to-video diffusion (19.8), and 16% for image-to-video diffusion with poke conditioning (20.4). A simulator oracle reaches 84% at 55,162.2 actions/min.
• Distributional heads matter. Ablating the posterior parametrization on OWM gives 0.110 for a GMM head, 0.033 for the flow-matching head without the scale cascade, and 0.029 with the cascade.
• Efficiency of the fused blocks. The fused attention blocks give roughly 2× faster sampling for a 32-timestep rollout (batch size 4, 16 trajectories), extending to roughly 3.7× at batch size 1.
• Uncertainty is calibrated above pixel level. The model's posterior standard deviation correlates with true error in log-log space once error exceeds roughly 1/512.
• Not reported. The paper content provided does not include per-method latency in seconds, training-set size for the video baselines, or the full appendix ablations referenced in Sections A and B.

Methodology in Plain English

The model takes one reference image and a small set of points the user cares about. Instead of generating pixels, it predicts how each point moves in the next short interval, then updates the points and repeats. This "one step at a time" loop mirrors the intuition that you cannot predict a billiard break in a single leap — you unroll it collision by collision.

Each predicted step is a small displacement (Δx_t^(i)) modeled with flow matching, a diffusion-style generative method. Every motion token mixes three things: image features sampled at the point's original location ("what" it is), image features at the point's current location ("where" it is now), and a random per-trajectory identity vector drawn from a unit sphere ("who" it is), which the authors say is critical in multi-trajectory settings. Motion tokens and image tokens share one positional encoding scheme based on axial RoPE.

For speed, the backbone uses parallel transformer blocks with a single residual, fusing the "up" projection (QKV plus feedforward up), the attention and feedforward outputs in the "down" projection, and combining self- and cross-attention in a prefix layout where image tokens attend to nothing and motion tokens attend causally to both streams. The flow-matching head is conditioned on cached representations, and a "scale cascade" feeds tanh-saturated, logarithmically spaced versions of the motion into the head, since motion values are heavy-tailed (the paper reports excess kurtosis in the hundreds). An optional initial motion "poke" conditions the rollout.

The model was trained on 10M open-set internet video clips with pseudo ground truth from TAPNext, plus 3D tracks from V-DPM projected to the first camera view for a smaller run (~1.5M clips), and separately on billiard simulation data for planning. Total trainable parameters are 665M, with an L-scale transformer and a DINOv3-L/16-initialized image encoder at 512² resolution.

Why This Matters

Impact on research. The paper argues that multi-step reasoning about scene motion does not require attending to every pixel, and backs this with a benchmark that measures distributional accuracy rather than point estimates. It reframes the world-modeling problem around dynamics rather than appearance, and shows that a step-wise, sparse approach can beat dense pixel-space simulators that have far more parameters.

Real-world applications (as suggested by the paper's settings):

• Robotics and manipulation, where many candidate action outcomes must be simulated quickly before choosing one.
• Sports and gameplay analysis, such as planning billiard shots to land a ball at a target location.
• Physical-reasoning evaluation, using the PhysicsIQ and Physion diagnostics to test whether models understand simple mechanics.
• Any forecasting setting where a domain expert can specify a handful of points of interest and ask "what happens next," rather than needing full frame generation.

Industry relevance. The efficiency claim is the practical hook: Myriad is listed at 2200 samples/min at 665M parameters, versus sub-1-sample-per-minute throughput for the largest video baselines, which the authors frame as the difference between exploring a few futures and exploring thousands within the same compute budget. An author is affiliated with Netflix, and the benchmark is slated for public release, making the setup directly usable for evaluation pipelines.

Future Directions

• Dynamic cameras. The main formulation assumes a static camera, which the authors state limits applicability to scenes with ego-motion or dynamic viewpoints — a setting that contemporary video generation baselines already handle. The paper mentions exploring learning from dynamic-camera videos by compensating during preprocessing, but says joint prediction of ego and scene motion remains future work.
• Broader planning tasks. The billiard experiment is one controlled planning domain; extending zero-shot action selection to more complex, real-world goals and reward structures is an open question.
• Scaling the data and representation. The 3D-track variant was trained on roughly 1.5M clips versus 10M for the main model because of tracker cost, so the trade-off between track quality and data scale is left open.
• Dependence on pseudo ground truth. The paper notes (in text truncated at "our model relies on pseud…") a limitation tied to pseudo ground-truth supervision; the provided content does not complete this point, so the exact concern is not reported here.

Target Audience

Researchers and practitioners in generative world models, video prediction, and trajectory forecasting who care about efficiency and multi-modal future prediction. It is also relevant to robotics and planning researchers who need fast rollouts of many candidate action outcomes, and to benchmark designers interested in evaluating distributions of trajectories rather than single point estimates.

Authors’ abstract

Accurately anticipating how complex, diverse scenes will evolve requires models that represent uncertainty, simulate along extended interaction chains, and efficiently explore many plausible futures. Yet most existing approaches rely on dense video or latent-space prediction, expending substantial capacity on dense appearance rather than on the underlying sparse trajectories of points in the scene. This makes large-scale exploration of future hypotheses costly and limits performance when long-horizon, multi-modal motion is essential. We address this by formulating the prediction of open-set future scene dynamics as step-wise inference over sparse point trajectories. Our autoregressive diffusion model advances these trajectories through short, locally predictable transitions, explicitly modeling the growth of uncertainty over time. This dynamics-centric representation enables fast rollout of thousands of diverse futures from a single image, optionally guided by initial constraints on motion, while maintaining physical plausibility and long-range coherence. We further introduce OWM, a benchmark for open-set motion prediction based on diverse in-the-wild videos, to evaluate accuracy and variability of predicted trajectory distributions under real-world uncertainty. Our method matches or surpasses dense simulators in predictive accuracy while achieving orders-of-magnitude higher sampling speed, making open-set future prediction both scalable and practical. Project page: http://compvis.github.io/myriad.