Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

AI summary

Overview

Research area: Computer Vision and Multimodal AI — specifically the physical grounding of video large multimodal models (video LMMs).

Technical level: Intermediate. Familiarity with LMMs, in-context learning, and basic grounding/localization concepts helps, but the paper's core argument is accessible to a general technical reader.

Scope: This paper introduces a large human-annotated dataset of hand-object contact and release events and uses it to show that state-of-the-art video LMMs can describe actions fluently while failing to physically ground where and when the interaction actually occurs.

What This Paper Is About

Video LMMs such as GPT, Gemini, and Qwen can narrate what is happening in a video — naming objects, describing hands and surfaces, and labeling actions — with increasing accuracy. The authors ask a sharper question: when an interaction physically begins or ends (a hand attaching to or detaching from an object), can these models identify the exact video frame and pixel location of that event? The paper argues that fluent action description is a form of "shortcut learning" that masks a fundamental failure to ground semantics in real physical events.

Key Contributions

1. CRID — the Contact-Release Interaction Dataset: A first-of-its-kind, large-scale dataset of 24,222 human-annotated core interaction events (13,816 contact, 10,406 release) across 10,130 videos drawn from Something-Something-V2, labeled by 250 Amazon Mechanical Turk annotators. Annotations include event type, agent-object pair type, and spatiotemporal location (frame number and 2D image coordinates).

2. A structured probing benchmark: A curated set of 99 short single-event sequences (plus matched non-event control clips) constructed from 33 source videos, designed to isolate one physical contact or release per clip so that frame-level detection can be scored fairly.

3. A systematic prompting study: Evaluation of five state-of-the-art LMMs (GPT-4o, GPT-5.2, Gemini-2.5-Flash, Gemini-3-Pro, Qwen-2.5VL-72B) under three in-context learning regimes (zero-shot, one-shot, two-shot) crossed with two modulating prompt conditions — Grounding (describe the scene, name the object) and Reasoning (chain-of-thought).

4. Identification of the "grounding gap": Evidence that semantic success (high action classification and object naming) coexists with near-chance physical event localization, and that adding reasoning or grounding prompts does not close that gap.

Main Findings

• Frame detection is near chance level: In the zero-shot regime, exact-frame detection of a contact/release event is roughly 8–12%, close to the 10% chance level expected for 10-frame clips. Allowing a one-frame tolerance raises the best model (Gemini-3-Pro) to about 42.9%.

• Errors are systematically ungrounded: A positive mean frame error (prediction before a contact, or after a release) often corresponds to frames where there is a clearly visible gap between hand and object — meaning the model predicts frames that contain no physical contact evidence at all.

• Models cannot say "no event": On non-event control clips containing hands and objects in motion but no contact, most models always produced a frame prediction with a plausible-sounding explanation. Only Qwen correctly identified the absence of an event, and only 18.2% of the time.

• Action and object understanding is strong: Across the evaluation set, most models exceed 85% Top-5 action-template accuracy and 70–97% object-naming accuracy. Gemini-2.5 led Top-1 action classification at 66.67%; GPT-5.2 and Gemini-3-Pro led object naming at ~97%. This confirms the failure is localization-specific, not a semantic or recognition deficit.

• Spatial localization is also poor: Using a 120×120 pixel box centered on the annotated event point, mean IoU was only 0.147 for Gemini-3-Pro and 0.086 for GPT-5.2.

• Reasoning and grounding prompts do not help: Contrary to prior work on chain-of-thought and visual grounding, the Reasoning and Grounding conditions either left performance unchanged or slightly decreased it across ICL regimes.

• Feedback-style ablation also fails: A two-shot variant with iterative numerical verification feedback (inspired by RL-style reasoning boosts) did not improve detection performance.

• Humans vastly outperform models: Two naive human annotators achieved 43.00% ± 5.04% exact-frame accuracy and 81.36% ± 4.18% one-frame-off accuracy, with 70.26% ± 6.93% accuracy on non-event clips and mean IoU of 0.71 ± 0.02 for event location. Inter-rater ICC was 0.87.

• Dataset annotation reliability: Inter-annotator agreement on CRID was high for the frame (ICC 0.95) and moderate for spatial location (ICC 0.73 for x, 0.39 for y; IoU 0.57 for a 120×120 box), indicating frame timing is more reliably annotated than precise pixel position.

• Interpretation as System 1 vs System 2: The authors frame the results using dual-process theory — models excel at System 1 intuitive pattern recognition ("there is a hand and a cup, so the action is picking up") but lack System 2 reasoning about discrete physical state changes such as attachment and detachment.

Methodology in Plain English

The researchers started from an existing video action dataset (Something-Something-V2) whose labels describe generic actions like "putting something into something." They recruited 250 crowd workers to watch these videos and mark the precise moment and location at which a hand or object either begins touching a target object (contact) or stops touching it (release). Each annotation recorded the event type, the kind of agent involved (hand, object, or surface), and both the frame number and pixel coordinates.

For the model experiments, they hand-picked 33 videos where individual events could be temporally isolated, and cropped 99 short 10-frame sequences, each containing exactly one contact or release. They also cropped matched control clips showing similar hands and objects but no actual physical event.

They then prompted five LMMs with the sequence of frames and asked a single question: in which frame does the interaction begin or end? They varied the experimental setup in three ways. First, in-context learning regime: zero-shot (no examples), one-shot (one worked example), and two-shot (two worked examples), with test performance averaged over many random example draws. Second, a Reasoning condition where the model was asked to produce step-by-step reasoning before answering. Third, a Grounding condition where the model first had to describe the video and name the target object. Correctness was scored with a frame-error tolerance (exact match or within one frame), and spatial accuracy was measured by IoU against a box centered on the human annotation.

To confirm the models weren't simply failing at basic perception, the authors also ran separate action-classification and object-naming tasks on the same videos, and finally had two naive human participants perform the identical frame-detection and localization tasks as a baseline.

Why This Matters

Impact on research: The paper challenges a common assumption in multimodal research — that strong performance on video QA and action recognition implies genuine physical understanding. It provides a reusable benchmark (CRID) and a concrete failure mode for the community to target, and it questions the widespread practice of adding chain-of-thought or grounding prompts as a remedy. The authors also argue that the limitation likely stems from loose integration between separately trained visual and language representations, which suggests architectural rather than prompting solutions.

Real-world applications:

• Robotics and manipulation: Robots that must decide when a grasp has succeeded or failed need precisely the contact/release detection that current LMMs lack; using LMMs as high-level planners without this grounding risks unreliable physical interaction.

• Assistive and AR/VR systems: Tools that track hands interacting with real objects (for tutoring, augmented instructions, or accessibility) require reliable event timing and localization, not just action labels.

• Video surveillance and safety monitoring: Detecting whether a person actually touched, handed over, or released an object is a physically grounded question that current models would answer unreliably.

• Content analysis and video indexing: Automatic tagging of fine-grained interaction moments for editing, search, or highlight extraction depends on frame-level grounding accuracy.

Industry relevance: Companies building embodied AI, robot foundation models, and video analytics pipelines rely on LMMs as perception backends. This paper shows that such models can supply semantically plausible but physically incorrect answers with high confidence, which is a meaningful reliability risk. It also motivates the annotation effort required to build the causal world models that would close the gap.

Future Directions

• Architectures with structured physical priors: The authors suggest that future models need causal world models or explicit representation of physical state changes (attached/detached) rather than treating interactions as pixel patterns.

• Motion-boundary attention: The paper explicitly recommends attending to motion and motion boundaries around the hand and object as a mechanism for detecting the exact instant of contact or release.

• Using CRID for training or fine-tuning: The dataset is released openly so that future models can learn from these scarce, fine-grained annotations and combine them with semantic knowledge, in the way humans do.

• Better spatial annotation and evaluation protocols: The moderate inter-annotator agreement on pixel coordinates (ICC 0.39 for y) and the low model IoU scores raise the question of how to define and benchmark event location more robustly.

• Reconciling System 1 and System 2 capabilities: An open question is whether scaling, better multimodal fusion, or explicit reasoning traces over discrete physical events can bridge the gap that prompting alone could not.

Target Audience

This paper is most valuable to multimodal and video-understanding researchers, especially those working on grounding, hallucination, and evaluation of LMMs. It is also relevant to roboticists and embodied AI practitioners who treat LMMs as perception modules, to dataset and benchmark builders interested in fine-grained physical annotation, and to cognitive-science-adjacent AI researchers interested in the System 1 / System 2 framing of model capabilities. Readers without a background in multimodal models will still follow the central argument, though the prompting-condition details and ICL terminology require some familiarity.

Authors’ abstract

Large multi-modal models (LMMs) show increasing performance in realistic visual tasks for images and, more recently, for videos. For example, given a video sequence, such models are able to describe in detail objects, the surroundings and dynamic actions. In this study, we explored the extent to which these models ground their semantic understanding in the actual visual input. Specifically, given sequences of hands interacting with objects, we asked models when and where the interaction begins or ends. For this purpose, we introduce a first of its kind, large-scale dataset with more than 20K annotated interactions on videos from the Something-Something-V2 dataset. 250 AMTurk human annotators labeled core interaction events, particularly when and where objects and agents become attached (`contact') or detached (`release'). We asked SoTA LMMs, including GPT, Gemini and Qwen to locate these events in short videos, each with a single event. The results show that while models reliably name target objects and identify actions, they exhibit a form of `shortcut learning' where semantic success masks a failure in physical grounding. Specifically, they consistently fail to identify the frame where the interaction begins or ends and poorly localize the physical event within the scene. This disconnect suggests that while LMMs excel at System 1 intuitive pattern recognition (naming the action and objects), they lack the System 2 cognitive foundations required to reason about physical primitives like `contact' and `release', hence truly ground dynamic scenes in physical reality.