PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

AI summary

Overview

Research area: Computer Vision / multimodal evaluation — specifically benchmarking physical intelligence in Vision-Language Models (VLMs).

Technical level: Intermediate. Readers benefit from familiarity with VLMs, video-based benchmarks, and standard evaluation metrics such as accuracy, mean IoU, SRCC/PLCC, and Micro-F1, but no specialist physics or engineering background is required.

Scope (one sentence): The paper introduces PhysVista, a benchmark that evaluates VLMs across a closed "seeing–reasoning–assessment" cognitive loop using both real-world and AI-generated videos.

Paper metadata: arXiv:2610.00559v1 [cs.CV], published 30 Sep 2026, licensed CC BY 4.0. Authors: Xinge Peng, Yiting Lu, Tianwu Zhi, Wen Wen, Jianzhao Liu, Xin Li, Zhibo Chen, with affiliations at the University of Science and Technology of China, ByteDance, and City University of Hong Kong. Code and data are linked at https://github.com/Helen1p/PhysVista.

What This Paper Is About

Current benchmarks test physical understanding in Vision-Language Models only at isolated stages — some measure perception, some measure reasoning, and they mostly ignore whether a model can actually judge how physically plausible a video is. The authors argue this fragmented approach cannot diagnose whether VLMs genuinely understand real-world dynamics, and that existing setups also suffer from language shortcuts, binary plausibility judgments, simulation-only data, and a near-total absence of AI-generated video content. PhysVista addresses this by formulating a unified, closed cognitive loop of physical state perception, physical dynamics reasoning, and physical plausibility assessment, evaluated over both real-world and generated videos.

Key Contributions

1. The PhysVista benchmark. A benchmark that restores the human "seeing–reasoning–assessment" cognitive loop for evaluating physical intelligence in VLMs, covering perception, reasoning, and plausibility evaluation using both real-world and AI-generated videos.
2. A versatile evaluation framework. The framework measures physical state perception, temporal and causal reasoning, and plausibility assessment, and explicitly separates event-level reasoning from scale-level reasoning (fine-grained differences in magnitude, quantity, position, or proportion).
3. Systematic task formulation and re-annotation. Rather than emphasizing new data collection, PhysVista focuses on principled task design, re-annotation of existing sources, fixed content-agnostic question templates, human verification, and rewriting of stylistically patterned samples to reduce language bias and video-blind answering.
4. Key insights into VLM physical intelligence. Detailed performance analysis across a diverse set of VLMs that documents the limitations of current models and informs the design of future physically grounded models.

Main Findings

• Perception outpaces reasoning and assessment. Models perform relatively better on perception tasks than on reasoning and assessment tasks, particularly on Spatial State Perception, yet remain weak on fine-grained perceptual capabilities such as Quantitative Scale Estimation and Physical Uncertainty Awareness.
• Physical Dynamics Reasoning is the hardest stage overall. The paper reports especially low accuracy on Physical Dynamics Prediction and Counterfactual Physical Reasoning.
• A clear cross-stage cognitive gap exists. The benchmark reveals substantial within-model inconsistencies in capability rankings: GPT-5.2 and Kimi K2.5 rank third and fourth on State Perception but drop substantially on Dynamics Reasoning, while Gemini 3.1 Pro and Doubao Seed 2.0 Pro rank third and fourth on Physical Dynamics Reasoning yet perform less strongly on Physical Plausibility Assessment.
• Strong semantic perception does not guarantee physical understanding. The authors conclude that high performance in observable state perception does not necessarily translate into equally strong causal reasoning, and that correct mechanism inference does not imply calibrated plausibility judgment.
• Temporal reasoning is easier than causal reasoning. Models generally perform better on Temporal Order Reconstruction than on causal reasoning tasks, suggesting temporal ordering can often be solved with basic physical commonsense.
• Prediction and counterfactual reasoning remain difficult. Physical Dynamics Prediction and Counterfactual Physical Reasoning require models to infer how dynamics evolve over time, and current models struggle to maintain both event-level correctness and fine-grained scale-level precision. Counterfactual reasoning is harder still because it additionally requires reasoning under hypothetical interventions.
• Physical Violation Critique is a relative bright spot. Reported scores include 88.00% for GPT-6 Sol and 87.00% for Claude Opus 5.5, while Violation detection measured by Micro-F1 peaks at 51.60% for Claude Opus 5.5.
• A systematic real-world versus generated gap. Across most tasks, accuracy on real-world videos exceeds accuracy on generated videos. The gap is largest for Physical Uncertainty Awareness and Physical Dynamics Prediction — tasks requiring abstract perception or fine-grained physical judgment that are sensitive to subtle physical inconsistencies in generated content.
• One notable exception. On Quantitative Scale Estimation, performance on generated videos slightly exceeds performance on real-world videos, which the authors attribute to possibly cleaner geometric layouts or more controlled object configurations.
• Score collapse in plausibility scoring. Several models — cited examples are Qwen3-VL-8B and Kimi K2.5 — concentrate a disproportionately large fraction of predictions on the highest score level for both generated and real-world videos, meaning correct predictions on real-world videos may arise merely from the default top score. Some models shift the default to the lowest level instead (cited examples: Gemini 3.1 Pro and Grok 4), which harms real-world performance.
• Competent scoring requires distributional separation. Models with genuine scoring ability show distinct distributions between generated and real-world videos: real-world predictions concentrate on high plausibility while generated predictions spread across lower levels.
• Comparison difficulty varies by pair type. Real-world vs. Generated is easiest, Generated vs. Generated is substantially harder, and Real-world vs. Real-world exposes over-confident preference — models tend to force a choice even when both videos are physically valid, indicating limited ability to recognize equivalence.

Methodology in Plain English

The authors build the benchmark around three stages that mirror how a person interprets a physical event.

Seeing (Physical State Perception). The model must recover observable physical states from video. This includes Spatial State Perception with four subtypes — Relationship (left/right, near/far, containment, support), Interaction (contact, collision), Scene Context (terrain, weather, surface properties), and Spatial Feasibility — plus Temporal Violation Localization (identifying the precise time interval of a physics violation, requiring second-level grounding), Camera Motion Recognition across ten predefined categories, Quantitative Scale Estimation (expressing relative object ratios as fractions such as 1/3 rather than descriptive adjectives), and Physical Uncertainty Awareness (Parameter Uncertainty about latent physical parameters such as friction coefficients or reflectance, and Outcome Uncertainty about inherently indeterminate futures).

Reasoning (Physical Dynamics Reasoning). The model must infer underlying physical processes. This stage covers Temporal Order Reconstruction from temporally shuffled frames sampled at irregular intervals, plus five causal subtasks: Physical Mechanism Reasoning, Physical Principle Violation Reasoning, Physical Dynamics Prediction, Counterfactual Physical Reasoning, and Physical Violation Critique.

Assessment (Physical Plausibility Assessment). The model assigns a 1-to-5 plausibility score to a single video, calibrated to the severity of inconsistencies, and ranks which of two videos is more physically plausible.

Data. Videos come from the real-world dataset WISA-80K, the video-generation dataset VideoPhy-2, and Multi-CamVideo (which provides camera pose trajectories). Cameras are selected for a single dominant motion with no interfering movement, and overly simple scenes are removed. Per-task sample counts are reported as: Spatial 200, Localization 200, Camera 200, Scale 60, Uncertainty 180; Order 150, Mechanism 192, Violation 224, Prediction 135, Counterfactual 119, Critique 100; Score 200, Comparison 100.

Question construction. Most tasks use one fixed, content-agnostic template with no slot filling, to minimize textual cues. Counterfactual conditions are introduced implicitly by modifying event scenarios rather than stating parameter changes. Option design follows four shared principles: all options are physically plausible, all align with the video context, the correct option has clearly superior causal explanatory power, and incorrect options are comparable in event-level structure or scale-level granularity.

Hierarchical option generation. For Physical Mechanism Reasoning and Counterfactual Physical Reasoning, the authors use a three-stage procedure: event-level generation of four coarse-grained base options; scale-level refinement of each into semantically similar variants; then either coarse-grained selection (top-ranked option per base group, testing event-level reasoning) or fine-grained selection (sampling variants within a base group, testing scale sensitivity).

Quality control. Tasks whose ground truth was produced using VLMs — such as Spatial State Perception and Parameter Uncertainty — were human-verified. Six human annotators worked on annotation and verification. Five independent human evaluators rated 40% of the benchmark dataset on a 1-to-5 scale for linguistic neutrality, question-answer precision, and physical reasoning validity. Samples answerable without viewing the video were removed, and samples with recognizable LLM-style phrasing were paraphrased.

Evaluation. Localization is measured with mean IoU, scoring with SRCC/PLCC, Violation with Micro-F1, and other tasks with accuracy. Average accuracy is task-count weighted. The paper compares GPT-6 Sol, GPT-5.2, Claude Opus 5.5, Claude Opus 4.6, Gemini 3.1 pro, Doubao Seed 2.0 Pro, Grok 4, Qwen3.5-Plus, Kimi K2.5, Qwen3.5-397B-A17B, Qwen3.5-122B-A10B, Qwen3.5-35B-A3B, Qwen3.5-27B, Qwen3-VL-30B-A3B, Qwen3-VL-32B, and Qwen3-VL-8B.

Why This Matters

Impact on research. PhysVista reframes physical intelligence evaluation from isolated capability testing to a closed loop, and argues for perception-driven evaluation grounded in visual observations rather than exam-style problem solving where visuals are auxiliary. It adds assessment — specifically 5-level scoring calibrated to violation severity and pairwise plausibility ranking — as the missing complement component to perception and reasoning. It also brings AI-generated video evaluation into the same framework as real-world video, which the authors identify as largely unexplored.

**

Authors’ abstract

Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities, yet whether they truly capture the physical consistency underlying real-world dynamics remains unclear. Existing benchmark paradigms often suffer from fragmented evaluation, focusing on isolated cognitive stages while overlooking the inherent synergy between perception, reasoning, and physical judgment. The lack of a holistic perspective limits the ability to diagnose whether VLMs can reliably evaluate the physical authenticity of emerging generative models. To address these issues, we introduce PhysVista, a benchmark designed to evaluate physical intelligence in VLMs through a closed cognitive loop framework inspired by the human seeing-reasoning-assessment process. PhysVista restores this loop by jointly evaluating physical state perception, physical dynamics reasoning, and physical plausibility assessment. It further distinguishes event-level reasoning and scale-level reasoning to enable fine-grained analysis of physical understanding. In addition, PhysVista incorporates both real-world and AI-generated videos, allowing evaluation across diverse domains and emerging generative scenarios. Extensive experiments across a diverse set of VLMs reveal substantial limitations in physical reasoning and plausibility assessment, highlighting a persistent gap between visual recognition and genuine physical understanding, and pointing toward more principled designs for physically grounded multimodal intelligence.