PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

AI summary

Overview

Research area: Evaluation and benchmarking of text-to-video (T2V) generation, specifically physical plausibility judged by vision-language models. Technical level: Intermediate. Scope: The paper introduces a scene-level diagnostic benchmark, PhyMetric, that replaces holistic "is this video physically plausible" scoring with 8,920 answer-balanced, scene-specific binary questions across 1,115 scenes, 50 physical categories, six physical domains and 12 evaluation dimensions, evaluated with four open-source T2V models and three VLM judges.

What This Paper Is About

Existing physical-plausibility benchmarks for generated video mostly produce a single holistic plausibility score or a broad quality dimension, which makes it unclear which physical behavior was actually tested and why a model failed. PhyMetric recasts the evaluation as verifiable, scene-grounded question answering: each scene gets eight targeted yes/no questions plus a structured physical prior describing the governing law, expected temporal sequence, observable signatures and common failure modes. The goal is to turn opaque holistic scoring into interpretable, scene-level physical diagnosis.

Key Contributions

1. A scene-level diagnostic benchmark. PhyMetric contains 1,115 scenes over 50 physical categories and six physical domains, paired with 8,920 answer-balanced binary questions spanning 12 evaluation dimensions. Every scene carries eight targeted yes/no questions and a structured physical prior.
2. A reformulation of physical evaluation. The paper recasts plausibility scoring as per-scene balanced binary QA grounded in a structured prior (governing law, expected temporal sequence, observable signatures, common failure modes), showing empirically that scene-specific questions are necessary for non-trivial evaluator accuracy while the prior provides a complementary, evaluator-dependent signal that improves human alignment.
3. Multi-model, multi-judge validation. PhyMetric is validated across four open-source T2V models (CogVideoX-5B, Wan2.1-1.3B, Wan2.2-14B, Cosmos-7B) and three VLM judges (Qwen3.5-9B, Qwen2.5-VL-7B-Instruct, GPT-5.4), showing consistent model rankings across judges, distinctive dimension-level capability profiles, and systematic weaknesses that holistic scoring does not surface.

Main Findings

• The benchmark separates current T2V models. On the full 1,115-scene benchmark, Wan2.2-14B has the best average score (72.78 Overall / 74.45 Weighted averaged across Qwen3.5-9B and Qwen2.5-VL-7B), followed closely by Cosmos-7B (72.59 / 74.26), then Wan2.1-1.3B (70.56 / 72.46), with CogVideoX-5B lowest (68.24 / 70.40). The paper notes the ranking is not explained by parameter count alone: CogVideoX-5B is larger than Wan2.1-1.3B but scores lower under both judges.
• The benchmark is not saturated. The paper reports that the gap between the best and weakest models shows current open-source T2V models are not saturated on PhyMetric.
• Judge behavior differs in level, not order. On the 100-scene controlled subset, Qwen2.5-7B is described as more conservative than Qwen3.5-9B and GPT-5.4, but the model ordering stays stable enough to support cross-evaluator comparison.
• Close overall scores hide different profiles. Cosmos-7B slightly exceeds Wan2.2-14B in the three-judge subset average (72.62 vs. 72.58 Overall), while Wan2.2-14B is stronger under Qwen3.5-9B, indicating that similar overall scores can mask different physical capability profiles.
• Dimension-level performance is very uneven. Across the 12 dimensions, average accuracy ranges from 90.39% on number down to 66.39% on appearance. Direction and motion both exceed 79%. The weakest dimensions are appearance, plus size and spatial, indicating persistent difficulty with material/state/surface consistency, object scale and relative placement over time.
• Dimension spread differs sharply by dimension. Appearance and shape show the largest inter-model spreads (10.00 and 10.53 points), while object identity shows only a 1.69-point spread.
• Scene-specific questions are essential. Replacing each targeted question with a single generic query ("Is this video physically plausible?") drops accuracy to near chance: Qwen3.5-9B falls from 72.75% to 48.13%, Qwen2.5-VL from 63.47% to 47.31%, and GPT-5.4 from 73.94% to 49.25%. Adding the structured prior to the generic query does not recover the lost signal (48.88%, 46.91% and 47.12% respectively).
• Structured priors give an evaluator-dependent benefit. Keeping scene-specific questions, adding the prior raises Qwen3.5-9B accuracy from 72.75% to 74.91% and its human correlation from 0.165 to 0.201; for Qwen2.5-VL accuracy rises from 63.47% to 66.09% but human correlation decreases; for GPT-5.4 label accuracy falls slightly (73.94% to 72.25%) while human correlation improves from 0.216 to 0.250. The paper concludes that label accuracy and human alignment measure related but distinct evaluator behaviors.
• Prior source is not just self-consistency. Across evaluators, the best human correlation often comes from a prior source different from the evaluator itself. For Qwen3.5-9B, DeepSeek-R1 and Mistral-7B priors reach the highest human correlations (ρ_human = 0.251 and 0.248). For GPT-5.4, every structured prior improves human correlation over zero-shot, with Mistral-7B reaching ρ_human = 0.293. Significance at p < 0.05 is marked only on selected correlations in the prior-source table.

Methodology in Plain English

PhyMetric is built in two layers.

Layer one: identifiable questions. Instead of one broad plausibility question per video, each scene gets a set of eight scene-specific binary questions written for the concrete objects, environment, action and expected physical consequences. The example used in the paper is tomato slicing: rather than asking whether the clip is plausible, PhyMetric asks whether the knife contacts the tomato, whether the tomato separates after the cut, and whether the two pieces remain visually consistent. Each question carries a ground-truth yes/no answer and one or more dimension tags. Because answers are balanced, 50% is a meaningful chance baseline. The scene score is the average of per-question correctness, and these scores are aggregated hierarchically into scene-, dimension-, category- and model-level scores.

Layer two: expectation grounding. Each scene is paired with a structured prior K_s = {ℓ, o, m, e, r, h}, where ℓ is the governing physical law, o the involved objects, m material or state information, e the expected event sequence, r likely failure modes and h dimension-level evaluation hints. The prior is built from the scene caption and category metadata alone, with no access to the generated video or the ground-truth answers. At evaluation time only {ℓ, e, r, h} are passed to the judge as textual context; {o, m} are auxiliary metadata used during construction.

When to look. Each question is tied to one of five temporal anchors (initial, interaction, transition, final, or full sequence). Anchor positions from a reference clip of N_ref frames are mapped linearly onto a generated video of N_V frames, and the mapped frame plus its neighbors form the evidence set handed to the judge. The paper states this linear mapping assumes the event's temporal location is approximately proportional to its location in the reference clip, and that this may fail under temporal compression or reordering.

Evaluation. A VLM judge answers each question from question-aware sampled frames, the scene caption and the prior. The setup uses four open-source T2V generators, each producing one video per scene from the original scene caption without model-specific prompt rewriting. Qwen3.5-9B is the default judge, with Qwen2.5-VL-7B-Instruct and GPT-5.4 used to test robustness. Results are reported as Overall accuracy (mean binary correctness over all questions) and Weighted accuracy (which incorporates the judge's self-reported confidence). Ablations and human validation use a 100-scene subset sampled as two scenes per category, annotated by 50 human annotators answering the same yes/no questions, with scene-level Spearman correlation ρ_human reported between VLM and human scores.

Why This Matters

Physical plausibility failures in generated video are easy to see and hard to attribute. Holistic scores can be high even when a key physical event never happens, and a low score gives no clue which physical law was violated. PhyMetric makes each score traceable to concrete question-level checks, which matters for both comparing models and diagnosing them.

• Model selection and procurement: the protocol yields stable rankings across three judges, which is what a buyer or integrator needs when choosing among open-source generators.
• Targeted model improvement: dimension-level profiles (for example weakness in appearance, size and spatial) point to specific failure classes rather than a single aggregate number.
• Evaluation of generated-media pipelines: video generation is increasingly used for dynamic-scene simulation, where the paper argues a clip must evolve according to the physical and causal regularities implied by the prompt and scene, not merely look realistic frame by frame.
• Human-aligned automated judging: the finding that structured physical priors improve human correlation for several judges is directly relevant to anyone building automated evaluators for video.

Industry relevance: the paper positions PhyMetric as both a comparison benchmark and a diagnostic framework for guiding future progress in physically grounded video generation. The ablation result that generic plausibility queries collapse to chance is a concrete warning against single-question VLM evaluation.

Future Directions

• Non-proportional event timing. The frame-sampling map assumes a generated event occurs at roughly the proportionally scaled position of the reference clip. The paper acknowledges this fails under temporal compression or reordering and defers the discussion to Appendix A.6, whose contents are not included in the provided material.
• Broader scope of the evaluative claim. The paper states that PhyMetric supports a focused claim about whether a generated video exhibits the specific physical consequences implied by the scene, and remains agnostic to broader questions of latent physical understanding or generalization beyond observed dynamics — leaving that broader question open.
• Prior quality across judges and sources. Because prior benefits vary by judge (improving accuracy for Qwen3.5-9B and Qwen2.5-VL but not GPT-5.4) and by prior source, the paper raises the open question of how to match priors to evaluators.
• Coverage and difficulty scaling. The paper describes its 50 categories as a coverage map over dynamics recurring in generated video rather than a topical list; extending coverage and difficulty is a natural next step.

Target Audience

Researchers and engineers working on text-to-video generation, video-generation benchmarking, and VLM-based evaluation; practitioners who need interpretable, attributable physical-plausibility measurements for generative video systems; and anyone studying whether automated judges align with human judgments of physical behavior in video.

Authors’ abstract

Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene. Existing benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection. Our project page is available at: https://mengsiwei.github.io/PhyMetric/.