VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

AI summary

Overview

Research area: Computer vision / video generation evaluation — specifically, physically grounded verification of AI-generated video and audio, with links to world models, embodied AI, and agentic tool use.

Technical level: Advanced. The paper assumes familiarity with video diffusion models (Wan 2.2, VACE-style control branches), flow-matching schedules, agentic reason–act–observe harnesses, segmentation and tracking models, and typed program validation.

Scope: The paper introduces VeriPhy, an auditable agentic critic that compiles a text prompt into typed physical obligations and a validated measurement plan before viewing any frame, then verifies generated audio-visual clips through frozen specialist tools and deterministic composition, evaluated against a 1,500-clip human-annotated flaw corpus.

What This Paper Is About

Generated videos can look convincing while violating basic physics: a ball passes through a wall, an object floats unsupported, an impact sound arrives before the collision. Existing evaluators usually return a single quality score or a yes/no question-decomposition answer, which does not say which obligation a clip violated, when it failed, or what evidence supported the judgment. VeriPhy aims to make every physical verdict traceable, by planning measurements before generating or observing video and by attaching a provenance-carrying evidence record to each decision.

Key Contributions

1. VeriPhy, an integrated physical-verification system. A text-only planner compiles an open prompt into typed physical claims and a statically validated execution plan before any frame is observed; observations then gate and scope only already-declared calls to frozen specialist operators; fixed composition maps usable evidence records to three-valued claim states (supported, contradicted, unknown), surfaced as plausible, implausible, or abstain.

2. A 1,500-clip corpus with 2,582 human-written, prompt-grounded flaw records, plus a flaw-level matching protocol that scores each system finding as whole, partial, or missed against the corresponding human record. Each record quotes the violated prompt span and provides rationale, severity, confidence, and, when available, a temporal span and object track.

3. A simulation-conditioned generation testbed. Known MuJoCo geometry is rendered as depth or silhouette control for a frozen Wan 2.2-VACE generator, separating a specified physical target from appearance so that control adherence and physical validity can be assessed separately.

4. A mechanism probe on control withdrawal. Using a separate Wan2.2-derived backbone without a trained control branch, the authors study when the control signal becomes redundant during sampling, reporting the withdrawal point as a relative noise level σ_rel rather than a step index.

Main Findings

• VeriPhy finds more annotated defects than the compared baseline. On a 149-clip core carrying 304 human-annotated flaw records, VeriPhy accounts for 228, compared with 164 for a published question-decomposition evaluator given the same clips, the same claims, and the same served model.

• Recall alone does not isolate the agentic design. A monolithic prompt to the same backbone reaches 222 of 304. The paper states that what separates VeriPhy is that each decision retains its evidence record and provenance, making traces auditable one verdict at a time and usable as an interface for writing a critic verdict back into generation.

• The reported numbers characterize the system, not its generalization. The core is a development set; the analysis is described as recall-only, single-annotator, and not held out.

• Evaluation and generation are decoupled. The paper reports that the current work evaluates the critic and the generator separately, and that feeding diagnoses back into regeneration remains future work.

• Prior physical evaluators are complementary, not uniformly weak. The paper argues that prior work already provides rule-level judgments, temporal localization, measurement-grounded tests, and audio or cross-modal consistency checks, but that these capabilities remain distributed across evaluators with different scopes, output contracts, and evidence representations. AV-Phys Agent is named as the closest point of comparison, though its released protocol stays tied to prompt-specific human-authored rubrics and multimodal-model binary judgments.

• Extra scaffolding is not automatically better. The related-work section cites controlled studies showing that additional tools or scaffolding may increase cost without improving accuracy, and the paper accordingly reports experience-conditioned harness execution rather than autonomous self-improvement or open-ended replanning.

• Timing is guarded, not trusted. Because a forced localizer may return its best candidate even for an absent event, event occurrence is first confirmed by the dense-frame semantic verifier (or on the audio track), and only supported occurrences let their intervals enter a temporal predicate. Timing may compose an order but never establish occurrence.

• No audio track yields unknown, not a negative measurement. A sound operation reading a clip without an audio track produces unknown rather than evidence against the claim.

Methodology in Plain English

VeriPhy works like a checklist that is written before anyone looks at the video. A planner that sees only the prompt (never the frames) breaks the prompt into short quoted spans, each tied to a checkable physical assertion, and assigns consistent names to entities and events. Claims are planned in groups of at most four against the full prompt, then renamed into one shared namespace so two groups cannot accidentally refer to the same object by different names.

Those surface checks are expanded into a typed dependency graph. Before anything runs, the graph must parse, have valid identifier references, be type-correct, be acyclic, and have every claim bound to at least one check. A language-model-written operation such as "check whether A precedes B" automatically pulls in the prerequisites it needs — occurrence gates and timing actions, plus co-visibility, mask, and depth actions for spatial relations. If a group cannot be planned after a fixed number of attempts, or if naming fails entirely, the claim falls back to direct semantic verification, and that fallback is recorded and charged as a model call rather than silently dropped.

Execution runs in topological order. Each action declares a scope — either the full frame set or a localized window derived from earlier results — and dispatches a frozen tool: SAM 3 for grounding and tracking, identity-based counting for cardinality, eleven track-based physical measurements, monocular depth, optical character recognition, and audio-event detection, alongside a video-aware semantic verifier for existence and event occurrence. Actions sharing an identical realized signature (clip, operation, arguments, references, and realized scope) are executed once. Every action returns a claim-bound evidence record carrying the realized scope, a measured/abstained/errored status, artifact hashes, cost, and provenance.

The eleven physical measurements share one grounding step and each corresponds to a class of human complaint. Displacement is the largest class and is reported in multiples of the subject's own radius, uses the furthest excursion rather than the net displacement, and removes camera motion via dense whole-frame optical flow. Other measurements cover contact (closest approach between tracked masks), shape change relative to background flow, path shape, permanence, hop counting, damping, onset ordering, clip timing, and viewpoint motion. An object under one percent of the frame or a track too short to measure triggers an explicit abstention, so an unmeasurable claim never resolves as supported. Only three sub-checks remain learned and are tagged as such: subtype identity, dense action-event recall, and depth-ordered spatial interpretation on localized evidence.

Verdicts come from fixed rules over usable records, not from a holistic model score. Temporal relations resolve to contradicted if either operand is contradicted, to unknown if either is unknown or a window is unavailable, and to an order predicate only when both events are supported. That predicate compares interval endpoints under a boundary slack of one quarter of the shorter window — a tolerance-scaled subset of Allen's interval algebra. The result is a three-valued claim state with a full trace linking it to the prompt span, planned calls, localized evidence, and composition rule.

For the generation side, a physics request ("a ball bounces once") goes through Author (compile the text into a scene specification with bodies, initial states, colliders, static surfaces, and a camera), Sim (run the physics solver, then validate the resulting trajectory against event-specific predicates), and Render (turn solver state into a depth control). If validation fails, the specification is revised and re-simulated for at most three revisions, i.e. four simulations; exhausting the budget yields an explicitly marked unvalidated fallback. The depth control normalizes camera distance using the 1st and 97th percentiles of non-background depths pooled over the whole clip, mapping to a near-bright, far-dark intensity with a small Gaussian blur; if that normalization fails, Render falls back to a binary moving-body silhouette. The control, an all-white generation mask, an appearance prompt, and a scalar strength condition a frozen Wan 2.2-VACE generator, where the control enters as a strength-scaled additive residual at each transformer layer rather than overwriting pixels.

Why This Matters

Impact on research. The paper argues that the gap in physical evaluation is not the absence of any single component but their integration: a verdict must establish that the relevant objects exist, localize contact windows, measure trajectory and geometry, and align audio events with visual windows, all on one shared clock. VeriPhy's contribution is a common verification contract — typed obligations, a statically validated plan, observation-scoped tool calls, measurements kept distinct from learned states, and fixed composition into three-valued decisions. Because every verdict retains its evidence record, the trace can serve as the interface through which a critic verdict could later be written back into generation.

Real-world applications:

• Screening robot-training trajectory data, where recent video world models have been used to synthesize trajectories and support policy learning and evaluation.
• Auditing generated media for physical implausibility in production pipelines, with localized timestamps and object tracks rather than a single score.
• Reducing the sim-to-real gap by verifying that simulated and synthesized motion obey contact and causal constraints.
• Building automated quality-control tools that must justify each rejection with reproducible evidence rather than a model's opinion.

Industry relevance. The system's design choices map onto practical constraints: tool calls are scoped to declared obligations, identical actions are executed once via common-subexpression reuse, fallbacks and abstentions are billed and logged rather than hidden, and terminal infrastructure failures are a separate outcome from an unknown claim. The paper also notes that additional tools or scaffolding can increase cost without improving accuracy, which is directly relevant to teams deciding how much agentic machinery to deploy.

Future Directions

• Feeding critic diagnoses back into regeneration, which the paper explicitly states remains future work; the current report evaluates the critic and generator separately.
• Automatic changes to tools, workflows, admission policies, and external knowledge, described as future work beyond experience-conditioned harness execution.
• Held-out, multi-annotator evaluation beyond the current development-set, recall-only, single-annotator analysis.
• Extending the control-withdrawal mechanism probe, which was run on a separate backbone rather than the deployed Wan 2.2-VACE generator, and read back as a property of the shared diffusion prior.

Target Audience

Researchers and engineers working on video generation evaluation, world models, and agentic verification systems; practitioners building physical-plausibility checks for embodied AI or robot-training data; and readers interested in how tool-using agents can be constrained so that every decision retains auditable evidence. The paper is written at an advanced technical level and assumes background in diffusion-based video generation, program-level plan validation, and multi-tool agent harnesses.

Authors’ abstract

Visual fluency in generated video does not imply physical reliability, and a scalar quality score alone is incapable of indicating the obligation a clip violates or the moment it fails. We present VeriPhy, an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations and a statically validated execution plan before any frame is observed. During execution, observations gate and scope only declared calls to frozen low-level experts (e.g., segmentation and tracking, counting, eleven typed physical measurements over the resulting tracks, depth, OCR, and audio-event detection). Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state. Typed resolvers and fixed composition map usable records to a three-valued state (supported, contradicted, or unknown, surfaced as plausible, implausible, or abstain) with full provenance, so that every verdict is traceable to the evidence that produced it. We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time. On a 149-clip core carrying 304 such records, VeriPhy accounts for 228, against 164 for a published question-decomposition evaluator given the same clips and the same claims. Recall alone does not separate it from prompting the same backbone monolithically, which reaches 222; what separates them is that each decision retains its evidence record and provenance, making the traces auditable one verdict at a time and usable as the interface through which a critic verdict could be written back into generation.