PhyRubric: Rubric-Grounded Diagnostic Evaluation of Physical Commonsense in Text-to-Video Generation

AI summary

Overview

Research area: Evaluation of text-to-video (T2V) generation, specifically physical commonsense reasoning and VLM-as-a-judge methodology (Computer Vision).

Technical level: Advanced — assumes familiarity with T2V models, vision-language models (VLMs), fine-tuning (LoRA, SFT/GRPO), and correlation/AUC-based judge evaluation.

Scope: The paper introduces PhyRubric, a rubric-based diagnostic evaluation framework, its benchmark (PhyRubric-Bench) and training set (PhyRubric-Train), and an open-source judge model (PhyRubric-Judge) trained to score and explain physical plausibility in generated videos.

What This Paper Is About

As text-to-video models are increasingly pitched as "world simulators," it matters whether their videos obey everyday physical expectations. Existing benchmarks ask human annotators to give each video a single holistic score, which produces ratings that are subjective (two raters can disagree on the same video) and uninterpretable (a score says nothing about which physical principle was broken). The paper's goal is to replace intuition-based scoring with a caption-specific rubric that is fixed before any video is seen, so every score is derived from item-level verdicts backed by timestamped evidence — and then to train an open-source judge that reproduces this diagnostic behavior.

Key Contributions

1. PhyRubric, a rubric-grounded evaluation framework. For each caption, a rubric is constructed before any video is generated and reused across all videos from that caption, decomposing it into semantic items (visible facts the caption requires) and physical items (which visible signs count as violations, each with a severity level and violation type). Scores are computed deterministically from item verdicts rather than rated directly.

2. PhyRubric-Bench, spanning 34 categories across five physical domains (mechanics, optics, heat, electricity, acoustics), with human-verified rubrics and human-verified labels. 245 captions are rendered by ten T2V models, yielding 2,433 videos (17 of the 10 × 245 planned videos were not produced because generators' safety filters declined the captions).

3. PhyRubric-Train and PhyRubric-Judge. 947 captions sharing the taxonomy but no scenes with the benchmark, labeled by a rubric-guided agentic annotator, produce 3,788 videos with 45.9K rubric-item verdicts. On these, the authors fine-tune PhyRubric-Judge, an open-source VLM (Qwen3-VL-8B) that needs no rubric at inference and outputs SA and PC scores plus a diagnosis.

4. An analysis-oriented evaluation showing that rubric-grounded scoring yields insights prior protocols miss — semantic adherence as the primary bottleneck, all-or-nothing physical errors, and a "avoidance" shortcut in which models score well on physics by simply not depicting the event.

Main Findings

• Physical commonsense is far from solved. Seedance 2.5 leads with an average joint performance (JP — the fraction of videos with SA ≥ 4 and PC ≥ 4) of 44.20%, followed by Veo 3.1 at 36.20%; the strongest open-source model, HunyuanVideo-1.5-8B, reaches 14.93%. Even the best proprietary model fails more than half of the time.

• Open-source models rarely depict the caption at all. Every open-source model has an average SA below 3, meaning at least one primary caption element is typically missing, and even among assessable videos their average PC never exceeds 3.5.

• Optics is easiest; heat and acoustics are hardest. Optics is the strongest domain for eight of ten models and second (or tied second) for the other two. Acoustics is worst: Seedance 2.5 reaches 30.00% JP and Veo 3.1 15.00%, and two open-source models score 0.00% in that domain.

• Semantic adherence is the main bottleneck. SA concentrates at score 2 (50.8% of all videos), while only 5.0% receive a 4. Most generated videos fail to depict the captioned event before physics even enters the picture.

• When the event is shown, physics is usually right. Only 5.9% of videos show the event fully but violate physics (SA ≥ 4, PC < 4), versus 16.9% that pass — 74.1% of fully shown events are rendered with plausible physics.

• Physical errors are all-or-nothing. PC over assessable videos is bimodal: 52.9% score 5 (no violation), 40.8% score 1 or 2 (fatal or core violation), and scores of 3 and 4 together account for only 6.3%.

• Models exploit a "don't show the event" shortcut. 25.0% of all videos never show the event. Avoidance rates run 20.0%–44.5% for open-source models versus 5.7% (Seedance 2.5) and 8.6% (Veo 3.1). Under prior protocols that score PC on every video, open-source PC rises by 0.16–0.59, and LTX-Video-2B (44.5% avoidance) would outrank Cosmos-Predict2.5-14B and Wan2.2-T2V-A14B; Cosmos-Predict2.5-2B would rank second on PC despite a JP of 6.69% (eighth of ten).

• PhyRubric-Judge matches a frontier proprietary judge. It reaches r = 0.61 / AUC 0.84 on SA and r = 0.32 / AUC 0.68 on PC, versus Gemini 4 Argon's r = 0.60 / AUC 0.84 and r = 0.31 / AUC 0.68. The best open-source baseline on SA is Qwen3-VL-32B at 0.49; on PC, VideoPhy-2-AutoEval reaches only 0.23 despite being fine-tuned on 62K human annotations, while PhyRubric-Judge is described in that comparison as trained on only 3,600 automatically labeled videos.

• The judge generalizes. On the six held-out generators it loses only 0.03 in SA and 0.06 in PC correlation and still beats every baseline. Training five leave-one-domain-out judges costs at most 0.04 in SA correlation and at most 0.09 in PC correlation (on mechanics).

• Failures are basic, not subtle. 41.6% of violations are L1 (core object principles), 50.7% L2 (everyday dynamics), and only 7.6% L3 (attribute experience). The two most frequent types, missing consequence (24.0%) and permanence & count (20.4%), account for nearly half. Examples: a spoon teleporting between 1.50 s and 1.62 s; a roof light already lit before the finger reaches the button at 2.30 s.

Methodology in Plain English

The researchers separate "what should happen" from "what did happen." For each caption, they write down — before generating or watching any video — a checklist: the visible facts the video must contain (semantic items, split into primary and secondary), and the physical expectations that could be broken (physical items). Each physical item has an Expect statement (where and when to look) and a Fail statement (the concrete visible signs that count as a violation), plus a severity — fatal, core, major, or minor — modeled on the order infants acquire physical knowledge, and one of ten violation types grouped into three levels.

An item is then marked pass, fail, or not shown (the relevant moment never appears — treated as absent, not as a physics failure). Scores follow from fixed arithmetic. SA: 5 if everything passes, 4 if only secondary items fail, 3 if exactly one primary item fails, 2 if two or more primary items fail. PC: the most severe failure sets a cap (fatal→1, core→2, major→3, minor→4), and each additional major-or-worse failure subtracts one, floored at 1. Rubrics are checked so every score from 1 to 5 is attainable, and each is human-reviewed once and then reused indefinitely — including for future generators.

To label data cheaply, an agentic annotator (built on Claude Sonnet 5.5) scans a timestamped overview of a clip and then zooms in at full frame rate where needed, returning verdicts with timestamped evidence. Benchmark labels are then human-verified; training labels are not, which is what makes 3,788 videos (45.9K verdicts) feasible. Finally, the authors fine-tune Qwen3-VL-8B-Instruct with LoRA (rank 16 on all linear layers of the language model, vision encoder and projector frozen) on caption plus 16 uniformly sampled frames at up to 448×448, for 2 epochs (900 steps), effective batch size 8, cosine schedule peaking at 1e-4. Training targets include not just scores but an assembled diagnosis, so the judge learns why, not only what.

Why This Matters

For research, the paper shows that how you score changes what you conclude. A single holistic rating hides the fact that models often dodge physics entirely by not rendering the event, and it conflates "wrong physics" with "didn't try." Deriving scores from auditable item verdicts makes benchmark results reproducible across raters, comparable across generators, and — crucially — actionable, since every low score names the violated expectation and its timestamp. The rubric-then-label-then-distill pipeline is also a template for building reliable open judges without proportional human labor.

Real-world applications:

• World-model and embodied-AI development: teams building simulators or robot policies need to know which physical law their generator breaks (permanence, solidity, causality) rather than a single quality number.
• Content production and VFX: automatic flagging of specific frames where an object teleports or a light turns on before the switch is pressed, with timestamps for re-rendering.
• Model selection and procurement: buyers comparing proprietary APIs and open checkpoints on a per-domain basis (e.g., optics vs. acoustics) instead of a single leaderboard rank.
• Training-data curation: the same rubric-driven verdicts can be used as reward or filter signals, not just evaluation.

Industry relevance is direct: the benchmark exposes the gap between frontier proprietary models (44.20% and 36.20% JP) and open-source ones (best 14.93%), and the released judge gives smaller teams a free approximation of a frontier proprietary evaluator (Gemini 4 Argon).

Future Directions

• Close the semantic adherence gap. With SA concentrated at score 2 for 50.8% of videos, the largest headroom is in rendering the caption's events and objects faithfully — the paper treats this as the bottleneck that physics evaluation sits behind.
• Attack the under-covered domains. Heat and acoustics are where even proprietary models collapse (30.00% and 15.00% JP on acoustics), and both require linking a visible trigger to an effect whose cause is invisible — an open modeling problem.
• Extend beyond five seconds and beyond single causal events. All videos are five to six seconds and every caption encodes one causal event with a stated outcome; whether these findings transfer to longer, multi-event, or open-ended generations is untested here.
• Reduce reliance on proprietary models in the labeling loop. The agentic annotator, rubric drafting, and caption generation all use proprietary LLMs; the paper's own judge is a step toward replacing them, but rubric construction for new captions still needs automation without a closed model.

Target Audience

Researchers and engineers working on text-to-video generation, video-generation evaluation, and VLM-as-a-judge systems; benchmark designers who need interpretable, reproducible scoring protocols; and practitioners selecting or debugging video generators who want per-domain, per-violation diagnostics rather than a single quality score. Readers should be comfortable with diffusion/video model terminology and standard correlation/AUC-based judge validation.

Authors’ abstract

As text-to-video (T2V) models are increasingly positioned as world simulators, it is essential to evaluate whether their videos conform to physical commonsense. However, existing benchmarks rely on annotators who rate each video with a single score based on intuition, yielding ratings that are subjective and do not tell which physical principle is violated. We introduce PhyRubric, a rubric-grounded diagnostic framework for evaluating physical commonsense in T2V generation.…