Improving the Physics of Video Generation with VJEPA-2 Reward Signal

AI summary

Overview

• Research area: Computer vision — video generation and self-supervised video representation learning (video world models, diffusion/autoregressive guidance, and physics plausibility evaluation).
• Technical level: Intermediate to Advanced. The core idea is intuitive, but the method is expressed through modified diffusion score functions and a self-supervised encoder/predictor architecture.
• Scope: A short technical report describing the winning entry of the PhysicsIQ Challenge (Perception Test Workshop, ICCV 2025), showing that a self-supervised video world model (VJEPA-2) can be used as a reward signal to make the MAGI-1 video generator produce more physically plausible videos.

What This Paper Is About

State-of-the-art video generative models produce visually realistic videos but have severely limited physical understanding, often generating implausible motion and interactions. Prior work on the Physics IQ benchmark showed that visual realism does not imply physics understanding, while separate work showed that intuitive physics understanding can emerge from self-supervised learning (SSL) pretraining on natural videos. This paper asks whether an SSL-based video world model can be harnessed to improve the physical plausibility of a generative video model.

Key Contributions

1. A method that couples the autoregressive video generative model MAGI-1 with VJEPA-2 (Video Joint Embedding Predictive Architecture 2), using VJEPA-2's surprise score as a reward signal during generation.
2. A modification to MAGI-1's inference-time score function that adds a term, weighted by ω_s, guiding generation toward lower surprise under the VJEPA-2 predictor.
3. A Best of N (BoN) selection scheme that draws 16 video samples per condition and keeps the one with the lowest average surprise score.
4. State-of-the-art PhysicsIQ results of 62.64 for video-to-video (V2V) generation and 36.86 for image-to-video (I2V) generation, improvements of +6.33 and +6.63 respectively over vanilla MAGI-1.

Main Findings

• Both generation settings improve: On the PhysicsIQ benchmark, MAGI-1 + VJEPA-2 reaches 62.64 on V2V (versus MAGI-1 at 56.31, a +6.33 gain) and 36.86 on I2V (versus MAGI-1 at 30.23, a +6.63 gain).
• The headline improvement is about 6%: The paper summarizes the overall gain as roughly ~6% in physics plausibility over the state-of-the-art video generative model.
• New state of the art on both tracks: On V2V, the compared baselines are Lumiere (23.00), VideoPoet (29.50) and MAGI-1 (56.31). On I2V, the compared baselines are Sora (10.00), Pika 1.0 (13.00), SVD (14.80), Lumiere (19.00), VideoPoet (20.30), Wan2.1 (20.89), Runway Gen 3 (22.80), CogVideoX (26.22) and MAGI-1 (30.23).
• Guidance and selection are combined: The reported numbers come from using VJEPA-2's surprise score both as a guidance term in the sampling score function and as the criterion for selecting among 16 samples per condition.
• Physics is a separable property from realism: The paper cites the Physics IQ benchmark's finding that visual realism does not imply physics understanding, motivating the need for a dedicated physics signal.
• Not reported: The paper does not report the size of the PhysicsIQ evaluation set, per-category breakdowns, ablations isolating guidance from BoN, compute costs, or human evaluations.

Methodology in Plain English

The authors take an existing autoregressive video generator, MAGI-1, which produces video chunk by chunk and can be conditioned on text, text plus image, or text plus video. At inference time, MAGI-1 steers its denoising with a score function that blends three signals: unconditioned generation, context (past chunks), and text conditioning, each controlled by a weight.

They then add a second model, VJEPA-2, a self-supervised video model with an encoder and a predictor trained on internet-scale data with a mask-denoising objective in representation space. VJEPA-2 is used as a kind of physics judge. Given the past video chunks, VJEPA-2's predictor produces what it expects to see next; the encoder embeds both that prediction and what MAGI-1 actually generated. The cosine similarity between the two embeddings is the "surprise score."

A low surprise score means the generator's output matches what the self-supervised model expects. The authors fold the negative gradient of this surprise score into MAGI-1's sampling score function with a new weight ω_s, pushing generation toward lower-surprise, more physically plausible videos. On top of that, they generate 16 candidate videos for each condition and use Best of N to keep the one with the lowest average surprise score. The result is evaluated with the PhysicsIQ metric on both video-to-video and image-to-video tasks.

Why This Matters

Impact on research. The work demonstrates that self-supervised video world models can serve as reusable reward or guidance signals for generative video models, an alternative to collecting human preference data or hand-crafted physics simulators. It also connects two previously separate lines of work: physics benchmarks that diagnose failures in generative models, and SSL pretraining that appears to learn intuitive physics. Since this is a short technical report, it primarily signals a direction rather than providing a deep mechanistic analysis.

Real-world applications:

• Content and entertainment: More physically believable generated video for film, advertising, and social media, where implausible object motion is a common artifact.
• Simulation and synthetic data: Physically plausible generated clips could be used as training or scenario data for robotics and embodied AI, where unrealistic dynamics would mislead downstream learners.
• Creative tools: Text-to-video, image-to-video, and video-to-video editing products where users expect objects to fall, collide, and persist consistently.
• Evaluation and quality control: Low surprise scores under a self-supervised world model could act as an automatic plausibility filter for large video generation pipelines.

Industry relevance. The work comes from FAIR, Meta Superintelligence Labs with academic collaborators, and targets a challenge run in conjunction with an ICCV workshop, indicating active interest from industrial video-generation teams. Using an existing self-supervised model as a reward signal is comparatively cheap compared with retraining a large generative model, making it an attractive post-hoc quality lever.

Future Directions

• Scaling the reward model: Testing whether larger or better self-supervised video world models yield larger physics gains than the observed ~6%.
• Scaling test-time compute: The method uses 16 samples per condition with Best of N; the paper does not explore how the gains change with more samples or with cheaper selection strategies.
• Separating guidance from selection: Ablating the ω_s guidance term against Best of N alone would clarify how much each component contributes, which this report does not report.
• Beyond physics benchmarks: Extending the approach to other video quality dimensions and to generative models other than MAGI-1, and to the broader set of tasks the PhysicsIQ benchmark covers.

Target Audience

Researchers and engineers working on video generation, diffusion/autoregressive sampling guidance, and self-supervised video representation learning. It is also relevant to practitioners looking for practical, test-time methods to improve the physical plausibility of existing video generators, and to readers following benchmark-driven evaluation of video models. Because it is a short technical report reporting a competition result, it is best suited to readers already comfortable with diffusion guidance and world-model architectures; newcomers will need additional background from the cited Physics IQ and VJEPA-2 papers.

Authors’ abstract

This is a short technical report describing the winning entry of the PhysicsIQ Challenge, presented at the Perception Test Workshop at ICCV 2025. State-of-the-art video generative models exhibit severely limited physical understanding, and often produce implausible videos. The Physics IQ benchmark has shown that visual realism does not imply physics understanding. Yet, intuitive physics understanding has shown to emerge from SSL pretraining on natural videos. In this report, we investigate whether we can leverage SSL-based video world models to improve the physics plausibility of video generative models. In particular, we build ontop of the state-of-the-art video generative model MAGI-1 and couple it with the recently introduced Video Joint Embedding Predictive Architecture 2 (VJEPA-2) to guide the generation process. We show that by leveraging VJEPA-2 as reward signal, we can improve the physics plausibility of state-of-the-art video generative models by ~6%.