Training Object Permanence in World Models

AI summary

Overview

Research area: Video generation models as world models, physical reasoning, and cognitive-science-inspired benchmarking (object permanence and object solidity).

Technical level: Intermediate. The paper combines a large-scale synthetic data pipeline (Blender), fine-tuning of a 16B video diffusion model, human pairwise preference evaluation, and a custom training stack, but the core concepts are explained in accessible cognitive-science terms.

Scope (one sentence): The paper introduces WROP, a 150-generator Blender benchmark and 1.5M-sample training corpus for object permanence and solidity in video-to-video world models, and shows that fine-tuning a 16B model on it produces the top-ranked true-continuation system across a 14-model blind human Elo study.

What This Paper Is About

Video generation models increasingly produce photorealistic, temporally coherent footage, yet they still let objects vanish behind occluders, reappear at impossible positions, or pass through solid barriers. Those failures concern object permanence (OP) and object solidity (OS), two components of what developmental scientists call core knowledge, present in human infants within the first months of life. The paper asks two questions: do current video models already exhibit object permanence and solidity, and if not, can these capacities be trained in with a dataset designed around cognitive-science task paradigms. To answer them, the authors build WROP, a benchmark and training resource of 150 hand-authored Blender task generators, and fine-tune a 16B world model called PWM-WROP on the resulting corpus.

Key Contributions

1. A cognitively grounded benchmark and training corpus. WROP comprises 150 hand-designed Blender generators organized into six task families (three probing object permanence, three probing object solidity), producing a 1.5-million-sample training corpus at 10,000 samples per generator and a fixed 300-question exam (two questions per generator). Every sample ships with an input video, a target video, a natural-language prompt, per-frame object trajectory arrays, and a scene-state metadata record.

2. A fine-tuned continuation model, PWM-WROP. The authors post-train a 16B world model from the Cosmos3-Nano base on the WROP corpus so that the key physical event and its consequence fall in the frames the model must generate. It ranks first among true-continuation models in a blind pairwise human study at Elo 1679.5, third overall.

3. An evaluation framework combining human and automatic measures. Human preference is collected as blind pairwise comparisons fitted with a Bradley–Terry model on the Elo scale, complemented by full-reference target-fit metrics (MSE, MAE, PSNR, SSIM, MS-SSIM, LPIPS, temporal-difference L1, final-frame variants, CLIP similarity, and FID) and task-level qualitative analysis.

4. A released training stack and engineering record. PWM, a native-PyTorch training stack for AWS Trainium2, is released with the checkpoint, along with the data, exam, model answers, scores, and weights. The stack shards the 36-layer model over a tensor-parallel × FSDP2 mesh of 64 NeuronCores and reaches 5.7 s per step at batch 16.

Main Findings

• Reference-to-video models top the overall leaderboard. Wan 3.0 Prime and MiniMax H3 tie for first at Elo 1723.6 with a 77.9% score rate over 52 games each; Seedance 2.5 ranks fourth at 1649.6. Because these systems regenerate the scene on their own timeline rather than extending the input from its final frame, they have more freedom to produce a physically coherent scene, though that freedom can also cause them to drift from the input's object identities and spatial arrangements.

• Fine-tuning on WROP data lifts a continuation model into the top three. PWM-WROP ranks third overall at Elo 1679.5 [1603.5, 1781.5] with a 73.1% score rate, leading the next-best true-continuation system, Grok Imagine (video extend, 1457.0), by 224 Elo points. LTX-2.3 Extend (1453.4) and MAGI-1 24B (1248.0) rank lower.

• Interface class explains the leaderboard better than model scale. Bootstrap top-1 probabilities are 46.8% for Wan 3.0 Prime, 36.4% for MiniMax H3, 12.3% for PWM-WROP, and 4.5% for Seedance 2.5; every other system has zero probability of ranking first. The five models from Runway Aleph 2 (1518.3) through Cosmos3 Super (1409.2) are separated by only 109 Elo points with overlapping intervals.

• PWM-WROP's advantage is uneven across the six task families. It ranks first in object-static occlusion (OP-2) and third in both Baillargeonian occlusion (OP-1) and container permanence (OP-3), but is more variable on solidity: second in object drop (OS-2), fifth in Baillargeonian obstruction (OS-1), and eighth in object collision (OS-3). The authors read this as evidence that OP and OS may require different internal representations, and that the current fine-tuning regime is more effective for occlusion tracking than for contact-based dynamics.

• The reference-to-video leaders draw their advantage largely from solidity. MiniMax H3 ranks first in both object drop and collision, winning 100% of its games in each, and Wan 3.0 Prime leads in container permanence and ranks second in collision, while their object-permanence rankings are comparatively moderate.

• Competitive ranking at low native resolution. PWM-WROP is evaluated at a native output resolution of 320 × 192, compared to 720p and 1080p outputs from competing systems. At matched resolution it obtains the best LPIPS and MS-SSIM against the target video.

• Human evaluation was tightly controlled. Twenty crowdsourced raters passed a qualification screen (threshold 8/10, median 9/10), answered 90.0% of embedded attention checks correctly, and agreed with themselves on 93.5% of repeated items (Cohen's κ = 0.891). No position bias was detected: the left-placed clip was preferred in 51.4% of non-tie judgments (p = 0.615). Of 480 scheduled judgments, 476 were completed, 361 of them between distinct models.

• Qualitative failures are task-specific. On G43 (OP-1), Gemini Omni Flash collapses all three balls into a tight cluster at the tunnel exit, violating lane identity and count, while PWM-WROP keeps each ball in its correct lane. On G19 (OP-2), Seedance 2.5 generates an oversized occluder over an empty region while the original objects remain visible, then suddenly materializes all three objects when the occluder moves away; PWM-WROP preserves and reveals the unchanged configuration. Seedance 2.5 also fails on G27 (OS-2).

Methodology in Plain English

The authors start from developmental psychology. Object permanence and object solidity are treated as the most primitive layer of core knowledge, so the benchmark is built to mirror classic violation-of-expectation paradigms rather than to test generic video quality. Each of the 150 tasks is authored as a self-contained, parameterized Blender scene specifying objects and their semantic roles, scene geometry, initial conditions, keyframed motion and contact events, camera configuration, a natural-language prompt, and the expected physical outcome. No rigid-body physics solver is used: trajectories are authored analytically so that occlusion, contact, and reappearance occur at controlled frames. Parameters are split into structural ones (object count, geometry, trajectory, occlusion configuration, aperture size, contact timing) that define the physical challenge and surface-level ones (color, material, lighting, camera viewpoint) that are randomized independently to prevent models from exploiting perceptual cues.

Every clip is 120 frames at 1280 × 720 and 24 fps, rendered with Blender 4.4.3 and EEVEE Next, and split at the onset of the key event into a 60-frame input video and a 60-frame target video. Generators run independently across parallel workers, retry failed renders automatically, and pass automated validation of the five-tuple package (input video, target video, prompt, trajectory, metadata) before admission; representative samples from every generator are then manually inspected.

For evaluation, a single inference harness passes each question's input video and prompt verbatim to all fourteen models, never exposing the target, and with no model-specific prompt engineering or server-side prompt expansion. The fourteen systems fall into three interface classes: true continuation (PWM-WROP, MAGI-1 24B, LTX-2.3 Extend, Grok Imagine), reference-to-video (Seedance 2.5, Wan 3.0 Prime, MiniMax H3), and edit/transfer (Wan-VACE 14B, HY-OmniWeaving, LTX-2.3 Dev, Cosmos3 Super, Kling O3 Pro, Gemini Omni Flash 1.1, Runway Aleph 2). Inputs shorter than a provider's minimum clip length are front-padded by repeating the first frame, and any returned source prefix is trimmed before evaluation. Human raters see the input video and prompt alongside two anonymized, randomly ordered continuations and choose A, B, or "about the same" on three joint criteria: alignment with the text description, natural motion and physical plausibility, and object permanence. Outcomes are aggregated with a Bradley–Terry model with ties scored as half-wins, reported on an Elo scale with mean 1500 and 95% confidence intervals from 1,000 rater-clustered bootstrap replicates.

PWM-WROP is fine-tuned from Cosmos3-Nano with the architecture and tokenizer unchanged, for one epoch on 1,500,000 samples using a 117-frame packed clip at 320 × 192 (57 conditioning frames followed by 60 predicted frames), with each sample's prompt as text conditioning. At inference it uses a fixed seed with UniPC, 35 steps, guidance 6.0, and shift 10.0; for the two 90-frame benchmark items, the first 33 input frames fall outside its conditioning window and are not seen.

Why This Matters

Impact on research. The paper gives the physical-reasoning community a benchmark whose scale, training split, and ground-truth authoring distinguish it from prior work. The authors argue existing investigations are limited to mostly two-dimensional environments, cover only image-to-video generation rather than video-to-video, and lack dedicated evaluation of object identity, physical constraints, and causal consequences; existing benchmarks also tend to have small per-task scale, minimal or absent training splits, and reliance on VLM-based scoring, which is unreliable precisely because multimodal language models show systematic core knowledge deficits. WROP's use of human raters against hand-authored ground truth sidesteps that problem, and the released corpus makes OP and OS trainable rather than only measurable.

Real-world applications:

• Video editing and post-production tools that must preserve object identity, count, and spatial layout across occlusions.
• Robotics and embodied simulation, where a model that permits interpenetration cannot produce physically valid collision or support-removal events.
• Autonomy and safety simulation, where reliable contact and containment dynamics underpin scenario generation.
• Content creation pipelines that need temporally coherent continuations of an existing clip rather than full scene regeneration.

Industry relevance. The leaderboard finds that interface class explains performance better than model scale, and that proprietary and open-weight systems land side by side in the middle of the ranking. Wan-VACE 14B performs similarly to other mid-ranked models despite its relatively small disclosed parameter count. The released native-PyTorch Trainium2 stack, with its documented path from 15.1 s to 5.7 s per step, offers a concrete engineering record for training large video world models on non-GPU hardware.

Future Directions

• Closing the OP/OS gap. PWM-WROP's family-level profile shows strong occlusion tracking but weak collision performance (eighth in object collision), suggesting that the two capacities may require different internal representations and that the current fine-tuning regime is better suited to permanence than to contact-based dynamics.

• Untangling architecture from training. The authors note explicitly that because the evaluated models differ in architecture, the performance gap between PWM-WROP and other continuation models cannot be attributed to training alone, leaving a controlled comparison as an open question.

• Pushing evaluation beyond trivially resolvable cases. The qualitative analysis identifies specific generators (G19, G27, G43) where failures are dramatic, and the exam is fixed at 300 questions, leaving room to expand the diagnostic resolution of the benchmark across its six families.

• Resolving the resolution confound. Automatic final-frame metrics are also computed at 1280 × 720, but resolution differences across providers remain a confound at that scale, and the paper's own headline model runs at 320 × 192.

Target Audience

Researchers and engineers working on video generation, world models, and embodied AI who need a controlled probe of physical reasoning; cognitive and developmental scientists interested in operationalizing core knowledge concepts for machine systems; and practitioners building video editing, continuation, or simulation pipelines who want to know which model classes actually preserve object identity and solidity. The paper is readable by graduate students and technically literate newcomers, though reproducing the training results requires substantial distributed-training infrastructure.

Note: the provided paper content is truncated mid-sentence within Section 5.2 (Qualitative Analysis), so the summary above covers only the material available up to that point. Sections following 5.2 are not reported here.

Authors’ abstract

Object permanence and solidity are hallmarks of human cognitive priors. Recent studies show that video generation models, a paradigmatic class of current world models, have begun to show emerged reasoning abilities, making them ideal candidates for building human-like physical intelligence. Do video models have emerged object permanence in them? If not, could we train them with a core-cognition inspired dataset? We introduce WROP (World Reasoning with Object Permanence), a data infrastructure of 150 hand-designed cognitive science inspired tasks, divided into six cognitive categories. We build Blender generators that randomize speed, lighting, camera angle, and other nuisance parameters while preserving each task's cognitive structure, yielding 10,000+ samples per task. We release a 1.5M-sample training corpus and a 300-question exam. On this exam we evaluate 14 video models: 3 reference-to-video, 7 edit, and 4 continuation, among which PWM-WROP, our 16B world model. In a blind pairwise Elo study, PWM-WROP ranks first among continuation models and third overall, behind only a statistical tie between two reference-to-video models. We release the data, exam, model answers, scores, weights, and PWM, our native-PyTorch training stack on AWS Trainium2.