PhyProbe: Rethinking Physical Consistency Evaluation in Generated Videos

AI summary

Overview

Research area: Computer Vision — evaluation of video generative models, specifically measuring physical consistency (physics plausibility) in generated videos. Technical level: Intermediate. Readers should know roughly how video diffusion/generative models work and what pairwise preference evaluation and rank correlation mean, but the method itself is conceptually simple. Scope: The paper proposes PhyProbe, a lightweight scoring head trained on top of a frozen spatio-temporal video encoder that outputs a continuous, calibrated "physical violation score" for a single generated video, and benchmarks it against VLM-based evaluators and reward models.

What This Paper Is About

Generated videos can look sharp and smoothly animated while still showing objects that float, pass through solids, or change shape — and existing evaluators (off-the-shelf VLMs or fine-tuned VLM evaluators trained on human ratings) either fail to notice these dynamics or overfit to one dataset's annotation style. The authors argue that physical consistency should be treated as a measurement problem: an evaluator must produce a continuous severity score on a stable, anchored scale, not just a yes/no plausibility judgment. PhyProbe addresses this by learning a scalar violation score from frozen video-encoder features using a mixture of ranking, regression, and anchoring supervision drawn from seven heterogeneous datasets.

Key Contributions

1. A representation-level evaluator instead of a language-mediated one. PhyProbe extracts features from a frozen pretrained spatio-temporal encoder and maps them through a small scoring head to a scalar violation score, avoiding VLM prompting or semantic reasoning about physics.
2. A unified training objective over heterogeneous supervision. The authors combine three signals in one loss: Bradley-Terry pairwise ranking, Huber regression on noisy scalar annotations, and an anchor/calibration loss that pushes clearly real videos toward 0 and clearly violated videos toward 1.
3. A curated multi-source training set of roughly 371K video pairs from seven sources (PAI-Bench 42K, VideoPhy2 19K, VideoFeedback2 31K, BrokenVideos 14K, ImpossibleVideos 27K, TRAVL 98K, Kinetics-700 140K), deliberately spanning real–generated and generated–generated pairs with image+text, text, or no semantic correspondence.
4. A parameter-efficient design: only about 1M trainable parameters on top of a frozen 2B-parameter Perception Encoder (PE-Core-G14-448), using statistics pooling (temporal mean, standard deviation, and max) plus a three-layer GELU MLP.

Main Findings

• Strong pairwise accuracy overall. PhyProbe reaches 98.0% on ImplausibleBench (image+text correspondence, real–generated), 66.1% on VideoPhy2 (text correspondence, generated–generated), 98.9% on PhyDetEx (no correspondence, real–generated), and 75.2% on VideoFeedback2 (no correspondence, generated–generated), for a weighted average of 82.4%. The paper states it has the highest pairwise accuracy on three of the four physical benchmarks, with 98.0–98.9% on real–generated pairs and 66.1–75.2% on harder generated–generated pairs.
• Largest gains where prior methods degrade. In the no-correspondence and generated–generated settings the baselines fall off sharply (e.g., VP2-AutoEval averages 32.1%, V-JEPA Surprise 46.6%, VideoScore2 59.9%, Gemini-3.1-Flash-Lite 77.3%, Gemini-3.1-Pro 78.1%), while PhyProbe holds up at 82.4% weighted average.
• Balanced human correlation across two annotation scales. PhyProbe gets Spearman ρ = 0.293 / Pearson r = 0.302 on VideoPhy2 and ρ = 0.596 / r = 0.604 on VideoFeedback2 (Fisher-z averaged 0.458 / 0.466). VP2-AutoEval is higher on VideoPhy2 (ρ = 0.359) but collapses on VideoFeedback2 (ρ = 0.235); VideoScore2 is the mirror image (ρ = 0.197 on VideoPhy2, 0.462 on VideoFeedback2), which the authors read as calibration to a single annotation distribution.
• Scores are continuous, not binned. Figure 5 shows prior evaluators cluster predictions around discrete human rating levels, whereas PhyProbe produces smoother, more separated score distributions with broader intra-class variation.
• Transfers to general preference benchmarks without preference labels. With a model-side tie threshold at the 20th percentile, PhyProbe scores 72.7% on MonetBench with ties and 71.0% without, above VisionReward (68.2% / 72.2%) and VideoReward (56.1% / 58.2%); it leads Rapidata-I2V at 62.7% and stays competitive on VideoGen-RewardBench (65.3% overall without ties). The authors hypothesize physics violations co-occur with broader quality degradations.
• Each loss term is necessary. Removing the ranking loss drops generated–generated accuracy on VideoPhy2 to 61.9%. Removing the magnitude regression leaves ordering largely intact but blows the score range to [-1.89, 3.45]. Removing the calibration loss collapses ImplausibleBench accuracy from 98.0% to 84.9% and expands the range to [-0.40, 4.13]. The full model produces a range of [-0.00, 1.15] before clipping.
• Backbone choice matters a lot. With Wan2.2 VAE (300M) the ImplausibleBench score is 71.7%; V-JEPA2-ViT-L/16 (300M) 76.1%; V-JEPA2-ViT-H/16 (600M) 81.4%; V-JEPA2-ViT-G/16 (1B) 88.6%; PE-Spatial-G14-448 (2B) 98.7%; PE-Core-G14-448 (2B) 98.0%. Weakly supervised Perception Encoder features are strongest overall; V-JEPA2 is competitive, while reconstruction-optimized latent representations transfer less well.
• Gains are not just the backbone. With an identical V-JEPA2 ViT-H/16 encoder, PhyProbe outperforms the V-JEPA Surprise baseline by 13.9–48.1 points on all four benchmarks.
• Pooling head ablation. Mean MLP: 0.36M params, 61.0% VideoPhy2 accuracy / ρ = 0.277, 73.2% VideoFeedback2 / ρ = 0.578. Attention pooling: 6.92M params (19.2×), 63.7% / 0.292 and 75.0% / 0.612. Statistics pooling (mean+std+max): 1.02M params (2.8×), 66.1% / 0.293 and 75.2% / 0.596 — the chosen trade-off.

Methodology in Plain English

The system has two parts. First, a large pretrained video encoder — the authors mainly use the 2B-parameter Perception Encoder — watches a clip and produces a feature for each frame; the encoder is frozen and never trained. Second, those frame features are summarized into one video-level vector by concatenating their average, standard deviation, and maximum over time, and a small three-layer network turns that vector into a single number: the predicted severity of physical violation. Because the summary is over statistics rather than a fixed number of frames, the score works for a single video of any length and needs no reference clip or matching prompt.

Training uses three kinds of signal at once. From pairs of videos the model learns ordering: "this clip is more physically broken than that one," with ties allowed for real–real pairs. From datasets with human ratings or derivable severity estimates (for instance, how large and long-lasting an annotated error is), the model learns magnitude through a robust regression loss. From anchor samples — high-confidence real videos labeled 0 and synthetic failure cases labeled 1 — the model learns scale, so that scores mean the same thing across datasets whose annotations were collected under different conventions. Data sources are mixed with dataset-level weights so large corpora do not dominate. At inference the raw output is clipped to [0, 1]; the authors stress that the clipping is a calibration target, not an architectural constraint, and that the unclipped distributions already stay close to that range.

Why This Matters

Research impact. The paper reframes physical consistency evaluation as measurement rather than classification, and shows that a frozen encoder plus a tiny head can beat 7B-parameter VLM evaluators on this axis. It also documents a concrete failure mode of annotation-fitted evaluators — calibration to one dataset's rating scale — and offers anchoring as a partial fix, which is directly relevant to how physics benchmarks are built and compared.

Real-world applications.

• Automatic quality filtering and ranking of generated video candidates in text-to-video and image-to-video pipelines, where a per-video scalar score is cheaper than human review.
• Training-time reward or rejection-sampling signal for video generation models that need to avoid implausible dynamics.
• Regression testing across model versions or checkpoints, using a fixed [0, 1] scale that does not drift as generators improve.
• Content moderation / QA in production video tools where physically impossible artifacts (object interpenetration, morphing, disappearing objects) signal a failed generation.

Industry relevance. The evaluator is cheap at inference (frozen backbone, ~1M trained parameters, no prompt or paired reference required) and it is evaluated on commercially relevant generators and benchmarks, which makes it practical for large-scale internal model evaluation rather than only academic leaderboards.

Future Directions

• Long-horizon temporal modeling. The authors note that current spatio-temporal representations capture long-range causal dependencies imperfectly, so violations that only emerge over extended time may be under-encoded; better temporal representations are an explicit open problem.
• Detecting brief or instantaneous violations. Sudden appearances, disappearances, and abrupt state changes may not persist long enough to register in the aggregated statistics — a failure mode the authors illustrate in Figure 8.
• Disentangling physics from general quality. Because human judgments conflate physical implausibility with rendering artifacts and overall quality, the authors call for a controlled benchmark that varies physical consistency while holding quality, scene, prompt, and generator fixed; they state that no benchmark with reliable human annotations currently enables this.
• Reducing false alarms on unusual-but-valid motion. PhyProbe sometimes assigns elevated violation scores to real videos with rare but physically valid dynamics, pointing to gaps in both the representation space and the training distribution.

Target Audience

Researchers and engineers working on video generation and video evaluation metrics; benchmark designers who need annotation-robust, cross-dataset scoring; practitioners building automated quality or reward pipelines for generative video; and readers interested in whether self-supervised or weakly supervised video representations can substitute for VLM-based reasoning on physically grounded tasks.

Authors’ abstract

Evaluating the physical consistency of generated videos remains a fundamental challenge. Existing approaches rely on off-the-shelf vision-language models, which can often be myopic to physical dynamics, or fine-tuned evaluators trained on human annotations, which overfit to dataset-specific cues and fail to generalize. A key challenge is that existing supervision sources provide either relative ordering or absolute scores, but not both reliably and consistently across varied settings. To this end, we introduce PhyProbe, an evaluator that extracts features from a frozen pretrained spatio-temporal encoder and maps them to a scalar physical consistency violation score via a lightweight scoring head. PhyProbe is trained through a unified objective combining pairwise ranking, regression on noisy scalar annotations, and anchor-based calibration over a curated set of heterogeneous supervision sources. Experiments show that PhyProbe outperforms prior methods on most pairwise benchmarks spanning real-generated and generated-generated pairs under varying correspondence, with the largest gains in no-correspondence and generated-generated settings where existing fine-tuned evaluators degrade sharply. PhyProbe achieves strong correlation with human judgments, with close agreement between rank-based and linear metrics, indicating that scores are both well ordered and anchored to a stable [0, 1] scale. Further, despite being trained on supervision indicative of physical consistency, without explicit general-preference labels, PhyProbe also performs competitively on human preference benchmarks: consistent with the observation that physics violations are entangled with broader quality degradations.