PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

AI summary

Overview

Research area: Computer vision / generative video models as world models; distributional evaluation of stochastic physical prediction.

Technical level: Advanced. The paper assumes familiarity with conditional distributions, total variation distance, sampling, LoRA fine-tuning, and video generation systems.

Scope: The paper formalizes "probabilistic alignment" as a distributional criterion for world models, introduces the PAWBench benchmark (50 scenarios, eight mechanism groups, eleven evaluated video generators) and the PAWEval outcome-level protocol, and probes whether prompting, noise sampling, or fine-tuning can improve alignment.

What This Paper Is About

Video generators are increasingly described as world models, but a real world model should reproduce the distribution of possible futures under the same initial observation and action, not just one plausible continuation. Existing benchmarks judge each generated video independently, so they cannot tell whether repeated generations recover the correct outcomes in the correct proportions. The paper asks how far current video generators are from this "probabilistically aligned" behavior, and tests whether language prompts, initial noise sampling, or model training can close the gap.

Key Contributions

1. Formalization of probabilistic alignment. The paper defines two levels: support alignment (the model can realize every distinct valid outcome, so supp(p_M) = Y) and probability-mass alignment (outcomes occur in the right proportions, so p_M(·|x,a) = q for a reference distribution q). A model can be diverse without being aligned.

2. PAWBench, a 50-scenario diagnostic benchmark. It contains 50 scenarios spanning eight mechanism groups, split into PAW-Calibration (25 scenarios with analytically specified or symmetry-derived reference distributions, such as tossing, rotation, routing, and draw-style randomizers) and PAW-Coverage (25 scenarios with enumerable but not reliably specifiable probabilities, covering collision, stability, agent interaction, and material transition). Source image and action prompt are held fixed across repeated rollouts.

3. PAWEval, an outcome-level evaluation protocol. It maps repeated video rollouts to terminal outcomes using scene-specific rubrics (applied by Gemini 3.5 Flash), records non-readable cases as outcome-readout failures, and aggregates labels into an empirical outcome distribution. PAW-Calibration compares this to the reference distribution using total variation distance (TVD); PAW-Coverage measures the fraction of valid outcomes observed.

4. Controlled probes at three intervention points. The paper tests language (VLM future sampling, prompt engineering, Oracle prompt engineering), initial noise sampling (Coupled to Control / C2C noise coupling), and model learning (five LoRA-adapted Wan2.2 models trained on different outcome ratios), separating inference-time steering and exploration from changes to the model's learned predictive distribution.

Main Findings

• No system meets all three requirements. Across the eleven evaluated models (HappyHorse, Veo3.1 Fast, Kling 3 Std., Seedance 2, Wan2.7, Wan2.2, LTX-2.3, LTX-2.5, Cosmos 3 Super I2V, LingBot-Video-MoE, MiniMax H3), none combines accurate outcome probabilities, broad valid-future coverage, and reliable scene-level performance. Cosmos 3 Super I2V has the lowest calibration TVD (20.5) but only 80.0% of its Calibration scenes pass the readout gate; LTX-2.3 has the highest Coverage average (71.7%), computed over only 72.0% of its Coverage scenes passing. Seedance 2 passes 100.0% of scenes on Calibration and LingBot-Video-MoE passes 100.0% on Coverage, but neither leads the corresponding conditional metric.

• Coverage and calibration are separate failures. Recovering broad support does not mean assigning the right probabilities, and this separation holds across physical mechanisms. PAW-Calibration captures probability misallocation (every rollouts may look plausible yet one outcome occurs far too often); PAW-Coverage captures missing outcomes (some valid futures never appear). The paper reports them separately rather than as one score.

• The gap is not a sampling artifact. Across the eleven generators, observed TVD averages 31.2. In Monte Carlo simulations that draw matched samples from the reference distributions while preserving each model's passing scenes and readable sample counts, average TVD stays below 9.22 in 99% of simulations. Increasing the rollout budget from K=1 to 100 raises coverage for three of four models but leaves calibration largely unchanged.

• PAWEval agrees with humans on clear cases. For the 888 videos where both PAWEval and the human panel (seven independent judgments per video, same scene-specific outcome space) give a clear terminal-outcome label, PAWEval agrees with the decisive human label on 722 (81.3%).

• Models do not track causal versus non-causal changes. With paired interventions, distributions shift incompletely or in the wrong direction under physically causal changes (a pencil tilt that alters the physical transition), while non-causal distractor text (Galton-board text) redirects probability mass even though the reference distribution is unchanged.

• Direct VLM sampling is already misaligned. Without being given target probabilities, the best VLM calibration TVD is 34.8 (GLM-5V Turbo) and the highest coverage is 46.6% (Gemini 3.5 Flash). The outcomes GPT-5.5 selects for prompt engineering score 44.3 Calibration TVD and 35.0% Coverage before any video is generated.

• Prompt engineering helps scene pass rates but not calibration. Passing GPT-5.5's selected outcomes to generators raises SPR for all four tested models, but raises Calibration TVD among passing scenes in every case and improves Coverage for only two. Oracle PE, which supplies target outcomes directly, lowers Calibration TVD and raises Coverage for every generator, yet generators realize only 37.6–58.1% of the requested outcomes.

• Coupled noise broadens exploration, not the learned distribution. C2C lowers mean Calibration TVD and raises mean Coverage for all three tested generators (Wan2.2, LTX-2.3, Cosmos 3 Super I2V), with gains varying by mechanism and no consistent SPR improvement. The paper reads this as broader exploration of existing possibilities rather than a change to what the model learned.

• Training mixtures provide only coarse control. Training five LoRA-adapted Wan2.2 models on left-fall shares from 0% to 100% shifts generated left-fall frequency in both the upright and left-leaning pencil scenes, and the relationship is nonlinear and not one-to-one. The upright scene's reference is 50/50 (Base TVD 17.3) and the left-leaning scene's is 100/0 (Base TVD 41.3). At 20% left-fall training data the model comes closest to the upright reference (TVD 23.5) while the left-leaning scene remains 50/50 (TVD 50.0); at 80% or 100%, the left-leaning scene reaches its 100/0 reference (TVD 0.0) but the upright pencil falls left almost every time (TVD 48.0 and 50.0 respectively). None of the five adapted models produces both references, because the same adjustment moves both scenes in the same direction.

Methodology in Plain English

The core idea is to stop grading videos one at a time and instead treat a generator as a random sampler. For each scenario the researchers fix one source image and one action prompt, then ask the model for the same thing many times (K=50 rollouts per scenario and system, with default inference settings). Because the inputs never change, any variation across rollouts must come from the model's own distribution over futures.

A scene-specific rubric then reads each rollout and assigns it to one terminal physical outcome (for example, Head or Tail), or flags it as an unreadable outcome-readout failure. Failures are reported separately because they are not physical outcomes and cannot count as extra possible futures. A scene passes the readout gate when no more than 30 of its 50 rollouts fail, meaning at least 20 yield readable, in-schema outcomes; Scene Pass Rate (SPR) is the share of the 25 scenes in a track that clear this gate.

Two regimes are scored. In PAW-Calibration, where a defensible reference distribution exists (derived from sector proportions, combinatorial counts, physical symmetry, or visible causal state, never from model outputs or a default uniform assumption), the empirical outcome distribution is compared to the reference by total variation distance — half the ℓ1 distance between two categorical distributions. In PAW-Coverage, where probabilities cannot be justified, the benchmark simply measures how much of the enumerable valid outcome set appears across rollouts.

Beyond measurement, the authors run interventions. Language: five VLMs are queried repeatedly under the same observation and action, and their text predictions are mapped into PAWBench's outcome space; GPT-5.5 also writes generator prompts naming a predicted future; and an Oracle condition hands the generator the exact target outcome to isolate "can it render what was asked for" from "does it pick the right distribution." Noise: a repulsive Gaussian coupling scheme (C2C) introduces negative dependence among the 50 initial-noise samples while preserving each sample's standard Gaussian marginal, so the budget and prompt stay matched. Training: five LoRA-adapted Wan2.2 models are trained on datasets with different left-fall proportions under a fixed recipe and budget, then evaluated on upright and left-leaning pencil scenes with a direction-neutral prompt.

Why This Matters

The paper's impact on research is to shift evaluation from per-video plausibility to distribution-level correctness, providing a formal criterion (support alignment and probability-mass alignment), a reusable benchmark, and an outcome-level readout protocol. It positions itself against physics-focused and video-quality benchmarks whose unit of analysis is an individual generation, and against CaliBench, which evaluates nine stochastic scenes with known reference distributions; PAWBench differs by splitting calibration from coverage and pairing measurement with controlled causal, language, sampling, and training probes. The authors describe the work as a foundation for future efforts toward probabilistically aligned world modeling, and note that interaction and planning depend on both the possible consequences of an action and their relative likelihoods.

Real-world applications implied by this framing:

• Robotics and embodied planning, where an agent must weigh several physically valid outcomes of the same action rather than assume one deterministic result.
• Simulation and decision support, where downstream choices depend on how likely each outcome is, not merely on whether an outcome is possible.
• Content and media generation, where users may want calibrated control over how often a generator produces each valid variant.
• Model evaluation and auditing, giving developers a diagnostic for whether a generator's stochastic behavior tracks physical state.

Industry relevance: the benchmark spans eleven proprietary and openly released systems from multiple organizations, and its interventions map onto three practical levers that developers already use — prompting, inference-time noise control, and fine-tuning. All three are shown to be insufficient on their own, which is directly relevant to anyone shipping or fine-tuning video generation models for prediction or control tasks.

Future Directions

• Move from terminal outcomes to richer state trajectories. The authors state that scoring only terminal outcomes makes distributional comparison tractable but misses trajectory-level dynamics and intermediate physical processes; they call for extending probabilistic alignment from terminal labels to richer state trajectories.
• Build efficient and reliable rollout-based estimators. Estimates rest on a finite number of rollouts; larger budgets reveal the induced distribution more reliably but raise evaluation cost and do not by themselves correct biased model distributions, so cheaper estimators are an open problem.
• Scale to interactive and embodied settings. PAWBench uses controlled, visually parseable scenarios; longer-horizon, interactive, and embodied environments remain for future study.
• Develop training objectives for calibrated distributions. The paper calls for objectives that explicitly learn calibrated distributions over possible futures across different physical states, since the LoRA experiment shows global frequency adjustment steers both scenes in the same direction and cannot deliver scene-conditioned alignment.

Target Audience

Researchers and engineers working on video generation, world models, model-based reinforcement learning, and generative-model evaluation will benefit most, particularly those interested in distributional metrics such as calibration and coverage. The paper is also relevant to practitioners who fine-tune or deploy video generators as predictive simulators, and to benchmark designers looking for a template that separates probability-mass alignment from valid-support recovery. Readers need a working grasp of conditional distributions, sampling, and total variation distance; beginners will find the conceptual framing (one plausible future is not enough) accessible but the metric definitions and intervention tables demanding.

Authors’ abstract

Recent video generation models are increasingly framed as world models. Many physical processes can unfold in more than one valid way. Therefore, a world model should reproduce not only a plausible trajectory, but also the distribution of possible behaviors under the same initial observation and action. We call this distribution-level requirement probabilistic alignment. However, existing evaluations largely assess individual-video plausibility and do not test whether repeated generations recover the correct distribution. This raises a central question: how far are current video generators from probabilistically aligned world modeling? To answer it, we formalize probabilistic alignment as a distributional criterion for world models and introduce PAWBench, a benchmark for evaluating video generators as stochastic samplers of world dynamics. We further introduce PAWEval, an outcome-level protocol that converts repeated video rollouts into empirical distributions over possible physical behaviors. Across 50 scenarios and eleven current systems, no model consistently matches the reference probabilities while recovering the range of valid behaviors. Having established this gap, we test whether language prompts, initial noise sampling, or model training can reshape the model's predictive distribution. We believe our work can serve as a foundation for future efforts to move towards probabilistically aligned world modeling.