WOVEN: Weaving Visual World Modeling into Multimodal LLMs

AI summary

Overview

Research area: Multimodal large language models (MLLMs) and visual world modeling — specifically whether reasoning about how a scene changes under an action can be taught as a single reusable skill.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with supervised fine-tuning, GRPO-style reinforcement learning, benchmark evaluation, and standard MLLM terminology.

Scope (one sentence): The paper builds WOVEN, a 36,076-item video-generated benchmark and training source organized by scene, action, and reasoning type, uses it to measure and then train visual transition reasoning in MLLMs, and derives a four-part recipe for choosing training supervision.

What This Paper Is About

MLLMs are strong at recognizing what is in an image but unreliable at reasoning about how a scene changes: they misjudge camera rotation, predict physically impossible outcomes, and misorder video frames. The authors argue these separate failures share one root cause — a deficit in visual transition reasoning, the ability to infer the missing part of an (initial state, action, resulting state) triplet. The goal is to test whether this single capability can be learned from one training source and reused across many downstream tasks, and to work out which kinds of supervision transfer best.

Key Contributions

1. A formalization and taxonomy of visual transition reasoning. The authors define the capability as inferring unobserved components of an (s, a, s′) transition and enumerate eight reasoning operations in four families: causal dynamics (forward dynamics, inverse dynamics), counterfactual reasoning (counterfactual removal, counterfactual substitution), physical modeling (outcome prediction, cued prediction), and temporal coherence (temporal ordering, temporal adjacency).

2. The WOVEN dataset and benchmark. 36,076 four-option multiple-choice items generated with the Wan2.2-I2V-A14B video model, spanning 20 scene types, 5 action types (one passive-physical type with 8 principles, and four agent-driven types: camera motion, object inspection, navigation, object manipulation), and the 8 reasoning types. Items carry error-typed distractors and are split into a 22,728-item training set, a 2,508-item validation set, and three test sets (6,228 in-distribution, 3,496 scene-OOD using held-out restaurant and beach scenes, and 1,116 state-perturbation).

3. Evidence that the deficit is systematic and learnable. Evaluation of 38 frontier MLLMs shows the best model, GPT-5.4, reaches 65.8% against a 92.3% human baseline, with the same failure patterns recurring across model families. Training Qwen2.5-VL-3B-Instruct on the full training set raises in-distribution accuracy from 26.4% to 89.3%.

4. A validated training recipe for visual world modeling. Through controlled subset comparisons and a prospective test on held-out benchmarks, the authors conclude that supervision should be selected by the reasoning operation it teaches rather than by its actions, scenes, or domains; that larger state changes give more robustness; that temporal and agent-driven tasks need their own supervision; and that non-transition capabilities such as static perception are unaffected.

Main Findings

• A large, structured gap in current models. Across 38 frontier MLLMs, even GPT-5.4 reaches only 65.8% in-distribution accuracy versus 92.3% for humans (mean of three annotators), and the next three strongest models fall between 57.8% and 63.2%. The gap persists with scale and recurs across families.

• The failures have a consistent shape. For every model, temporal adjacency is harder than temporal ordering (25.3%–28.0% versus 30.8%–63.4%), counterfactual removal is harder than counterfactual substitution (54.8%–66.7% versus 64.5%–76.0%), and accuracy drops more under geometric than appearance perturbations (33.3%–62.7% versus 79.2%–95.5%).

• The capability is learnable and data-efficient. Full-set training takes Qwen2.5-VL-3B-Instruct from 26.4% to 89.3% in distribution and 88.2% on held-out scenes. With roughly 7 hours of generated video, the post-training gain on a spatial-reasoning benchmark exceeds that of Orca, which mid-trains the same backbone on 125,000 hours of video.

• Transfer is broad. Training on the full set improves 12 of 26 external benchmarks, with gains up to 24.0 percentage points on SAT. Controlled subsets of about 2,000 items each, choosing the best subset per benchmark, collectively improve 22 of 26 benchmarks — every benchmark involving visual transitions — with gains up to 27.3 percentage points. The 4 static-perception and general video-QA benchmarks do not improve under any training.

• The gains behave like one shared skill, not many separate ones. Every subset improves WOVEN accuracy even on action types it never trains on, and forward-prediction training alone raises counterfactual substitution and removal by 48 and 25 percentage points. The correlation between a subset's WOVEN improvement and its average external gain is r = 0.86.

• Transition supervision can substitute for task-specific data. Under a fixed 2,000-item budget, replacing 30–50% of a benchmark's own training data with WOVEN items preserves comparable accuracy across SAT, ActionEQA, and CLEVRER over three seeds; on SAT, training exclusively on WOVEN gives the highest accuracy in the sweep.

• Reasoning operation, not action or scene, predicts transfer. After removing the shared component via per-benchmark z-scoring, subsets teaching the same reasoning family have a mean pairwise residual correlation of +0.32, while subsets sharing only an action type are no more similar than subsets sharing neither (−0.22 versus −0.23).

• Larger state changes produce greater robustness. Robustness gains rise from camera motion (+1.7) and object inspection (+3.0) to navigation (+13.6), object manipulation (+23.2), and passive physical events (+29.9).

• Transfer has hard boundaries. The four temporal subsets lift temporal-adjacency accuracy from 25.7% to 55.6–58.7%, while the other seven subsets leave it near baseline, and ordering and adjacency supervision do not transfer to each other. The passive-physics subset, which contains no agent actions, actually lowers WorldPrediction by 10.2 and ActionEQA by 1.7 points.

• The recipe predicts transfer on unseen benchmarks. In a prospective test over 8 models on benchmarks not used to derive the recipe, the recipe-selected mixture beat alternatives by 1.5 to 4.7 percentage points on agent-action questions (p ≤ 0.03) and by 0.6 points on camera-motion questions (p = 0.04); the one predicted loss, on passive-physics questions against a passive-physics-heavy mixture, also appeared (−0.5, p = 0.04).

Methodology in Plain English

The authors start from the observation that many different-sounding tasks — predicting what happens next, inferring what action caused a change, imagining an alternative outcome, ordering shuffled frames — are all questions about the same triple: a scene, an action, and the resulting scene.

Because real video cannot supply controlled actions or alternative outcomes from the same starting frame, and simulators lack realism and diversity, they use a video-generation model, Wan2.2-I2V-A14B, to roll out transitions from specified starting frames and action descriptions. Two design rules keep the supervision honest: the resulting and intermediate frames are produced by the generator rather than written into the prompt, and multiple rollouts from the same starting frame under different actions supply counterfactual outcomes.

Each rollout becomes a multiple-choice question in which one part of the transition is hidden and the model must recover it. Thirty randomly chosen question templates keep the wording varied. Wrong options are taken from other rollouts of the same starting state and labeled with the specific error they embody (for example, a reversed direction or a gravity violation), so that failures can be attributed to particular mistakes rather than lumped into one accuracy number.

With this dataset the authors run three kinds of experiment. First, they test 38 existing MLLMs to measure the deficit. Second, they fine-tune models of different scales on the full training set, and separately on 11 controlled subsets of roughly 2,000 items each, each defined by one action type and one reasoning family, then evaluate zero-shot on 26 external benchmarks. Third, they run controlled comparisons — swapping subsets, replacing portions of a benchmark's own training data, and correlating transfer profiles — to isolate what drives transfer. Finally, they use the resulting recipe to pick training mixtures before seeing results, and check the predictions on benchmarks that played no role in building the recipe.

Why This Matters

The paper reframes a scattered set of MLLM weaknesses as one trainable capability, and shows the capability can be taught cheaply — thousands of generated items, not hundreds of thousands of hours of video. That is a practical alternative to scaling data for every downstream task separately, and the controlled design turns an intuition ("similar data transfers") into a testable, falsifiable claim about what actually drives transfer.

Real-world applications (implied by the capability, not claimed as deployments in the paper):

• Robotics and embodied agents, where a policy must predict the consequence of an action before committing to it, and where the paper's embodied benchmarks (ActionEQA, Robo2VLM, ERQA) are direct proxies.
• Video generation and editing pipelines, where a model needs to judge whether a generated clip respects physical constraints, causal order, and temporal continuity.
• Assistive and instructional systems that must answer "what happens if I do this?" questions about a scene — assembly guidance, cooking, or procedure tutoring.
• Spatial and physical reasoning in multimodal assistants, for example judging viewpoint changes from camera motion or anticipating collisions from a video feed.

Industry relevance: The finding that a small, generated dataset can replace 30–50% of task-specific training data is directly relevant to teams paying for domain-specific annotation, and the recipe gives a concrete selection rule for building training mixtures rather than pooling whatever data is on hand.

Future Directions

• Closing the temporal and agent-driven boundaries. Transfer does not cross from the temporal family to the other three, nor from passive physical events to agent-driven actions; the paper documents these walls but does not remove them.
• Extending beyond state transitions. Static perception and general video QA did not improve under any training, so the authors conclude those capabilities need their own supervision — how to combine transition supervision with other primitives is left open.
• Scaling the study. Training is demonstrated on Qwen2.5-VL-3B-Instruct with scale and cross-family checks reported in appendices; whether the recipe holds for the largest frontier models, and how it interacts with much larger backbones, remains to be tested.
• Broadening the reasoning taxonomy and the generator. The work covers eight reasoning operations generated by one video model (Wan2.2-I2V-A14B); adding operations and testing sensitivity to generator quality and artifacts are natural extensions, and the paper notes that real video offers neither controlled actions nor alternative outcomes from the same starting state.

Target Audience

Researchers and engineers working on multimodal LLMs, visual world models, and embodied AI who need either a diagnostic benchmark for transition reasoning or a practical rule for choosing post-training data. It is also useful for practitioners doing data-efficient fine-tuning, since the substitution experiments and the "select by reasoning operation" recipe translate directly into data-curation decisions. Readers wanting a first look at world-model training for MLLMs will find the taxonomy and controlled-comparison design accessible, though the training sections assume comfort with SFT and GRPO.

Authors’ abstract

Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from different supervision sources and reuse across different tasks, with a systematic training recipe. Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. We therefore introduce WOVEN, a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types. We first evaluate 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) and find a substantial and systematic deficit: even the strongest models fall far below humans, and the failures recur across model families and persist with scale. We then train MLLMs at multiple scales on WOVEN and find that they learn a shared capability that transfers broadly: training subsets of only about 2,000 items each collectively improve 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data can replace 30-50% of a task's own training data with comparable accuracy. Controlled comparisons further yield a training recipe for visual world modeling, validated prospectively on held-out benchmarks: select supervision by the reasoning operation it teaches rather than by the actions, scenes, or domains it shows, and prefer larger changes to the visual state for robustness. Our work establishes visual transition reasoning as a reusable foundation for systematic visual world-model training in MLLMs.