HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies

AI summary

Overview

Research area: Computer vision and multimodal video understanding, specifically human-centric physical reasoning, video moment retrieval, and human-scene contact estimation.

Technical level: Intermediate. The paper assumes familiarity with video moment retrieval metrics (mAP, Recall@1, tIoU) and SMPL-X body meshes, but the core idea is easy to grasp.

Scope: The paper defines a new task, human-centric impact localization, and introduces HAPACT, a movie-based benchmark of 2,884 video-query pairs with 20,369 annotated frames for localizing physical impacts in time and on the human body.

What This Paper Is About

Existing body-centric physical interaction datasets mostly capture everyday, user-initiated interactions such as contact, pressure, pose, and motion. Short, forceful, externally initiated events, such as a person being struck, falling, or hit by an object, are largely missing from these benchmarks. The authors define the task of localizing a queried physical impact both temporally within a video and spatially on the human body, and build HAPACT from movie clips to support and measure progress on that task.

Key Contributions

1. A new task. The authors formulate human-centric impact localization, which requires identifying the video frames in which a queried physical impact occurs and predicting the affected region of the human body.
2. A new benchmark. HAPACT is described as the first human-centric video benchmark for physical impact events, with viewer-grounded, text-conditioned annotations and frame-level SMPL-X labels, built from over 5 hours of video spanning 212 movies.
3. Systematic evaluation of existing methods. The benchmark evaluates 18 temporal baselines (8 retrieval-based models and 10 generation-based models spanning VMR-specific models, open-source MLLMs, and closed-source MLLMs) and three human-scene contact models (BSTRO, DECO, GraphiContact) for spatial localization.
4. A text-conditioned spatial baseline. Since contact models do not take text input, the authors reformulate on-body localization as query-guided body-region selection and instantiate it by fine-tuning the Cosmos-Reason2 MLLM over 18 candidate body region groups.

Main Findings

• Retrieval models need in-domain training. Zero-shot QVHighlights checkpoints do not transfer consistently: several retrieval models stay close to the random baseline (random mAP@0.3 of 4.14). Fine-tuning on HAPACT improves every retrieval model, with Sim-DETR reaching the best retrieval result of 23.27 mAP@0.3 and 26.07 G-mAP, but scores remain moderate.
• General MLLMs are the strongest temporal baselines. GPT-5.4 achieves 42.64 mAP@0.3 and 43.64 G-mAP zero-shot, ahead of GPT-4o (28.95) and Gemini 3.1 Pro (27.22). VMR-specific generation models are far weaker, for example VTimeLLM at 1.04 mAP@0.3 and UniTime at 7.93.
• Multi-moment reasoning is a clear weakness. Even GPT-5.4 falls from 54.49 mAP@1_tgt to 26.90 mAP@2_tgt and 14.74 mAP@3+tgt, declines of about 51% and 73%.
• Short impacts are hardest. Across representative models, moments of 1–2 frames consistently yield lower mAP than middle (3–6 frame) and long (7+ frame) moments, consistent with the dataset's average moment length of 4.80 frames, roughly 0.20 seconds at 24 FPS.
• Contact models transfer poorly to impact. After fine-tuning, vertex-level F1 stays below 9% for all three human-scene contact models (GraphiContact 7.34, BSTRO 5.72, DECO 5.41), indicating that image-only contact cues do not reliably identify impact-affected regions.
• Coarse region selection works better but is uneven. Cosmos-Reason2-8B with region selection obtains the best spatial scores (F1 18.59, precision 16.17, recall 44.11, geodesic error 33.31 cm), ahead of the 2B variant (F1 13.16). Its label-level F1 concentrates on the head, back, and buttocks, with many regions remaining low even after balancing the label distribution.
• Impacts cluster on specific body parts. Across the dataset, the head shows the highest impact intensity (23.3%), followed by the hips (8.4%) and spine (6.9%), with frequent impacts on the back, buttocks, and palms.
• The candidate pipeline has high recall. Author re-annotation of the source videos found a mean recall of 88.99% (95% CI 87.98–90.00%) for the vision-language-model-plus-filter candidate generation stage.

Methodology in Plain English

The authors start from movie clips on the Movieclips YouTube channel and parse movie titles and IMDb metadata, excluding TV shows, documentaries, and adult-rated films. Because impacts are not evenly spread across genres, they build a genre prior from films screened in 4DX format, scoring each movie by how often its individual genres and genre pairs appear among those films, and keeping movies scoring at least 0.1.

Candidate moments are generated by prompting Gemini 2.5 Pro to return second-level intervals and short descriptions of impacts on a person. Each interval is expanded by a 2.5-second buffer on both sides, and candidates longer than 8 seconds are dropped. Two filters then keep only human-targeted impacts: GPT-4o judges whether a description physically involves a human, and a Stanford Parser-based filter checks whether the affected argument denotes a person; the union of the two is kept. Author pairs then manually review candidates for a single visible, physically plausible target person, editing queries where needed.

Annotation happens in two stages using the HumanSignal platform with 28 trained annotators selected from 52 applicants, over roughly 30 weeks. In stage 1, annotators mark frame-level start and end frames for every occurrence of the queried impact on clips re-encoded at 24 FPS; three independent annotations are collected per pair and combined by majority vote, with ambiguous cases adjudicated manually. In stage 2, annotators paint the affected regions directly on an SMPL-X mesh per frame using a custom web-based painting interface, exporting vertex-level labels across frames. Splits are movie-disjoint at 70:15:15 to avoid content leakage: 2,019 training pairs with 14,264 annotated frames, 432 validation pairs with 2,963 frames, and 433 test pairs with 3,142 frames.

For evaluation, temporal localization is scored with mAP and Recall@1 at a tIoU threshold of 0.3, plus G-mAP and mean recall over tIoU thresholds from 0.1 to 0.5 in steps of 0.1, broken down by single-moment, two-moment, and three-or-more-moment queries. Spatial localization is scored with frame-wise precision, recall, F1, and geodesic error in centimeters, plus per-region F1 over seven merged body region groups.

Why This Matters

The work extends video-based physical reasoning beyond user-initiated interactions to externally driven bodily impacts, giving the field a measurement target for events that are brief, forceful, and caused by other agents or the environment. It also shows concretely where current architectures break: contact estimators trained for sustained contact, and moment retrieval models tuned for longer moments, both degrade on this task.

Real-world applications include:

• Haptic authoring and automated haptic experience generation for audiovisual content, where knowing when and where an impact lands is a prerequisite for rendering a tactile cue.
• Immersive media and mixed reality experiences that react to on-screen physical events.
• Body-aware video analysis, such as tagging or moderating violent content.
• Physical event analysis in media production, including VFX and stunt review.

Industry relevance is strongest for entertainment and XR pipelines that need event-level, body-part-level metadata from footage, and for teams building multimodal video models that must reason about force rather than just actions. The authors note the dataset does not redistribute the underlying copyrighted footage, providing only source identifiers, timestamps, IMDb identifiers, annotations, and evaluation scripts, and they caution that the clips include violence, falls, collisions, and explosions.

Future Directions

• Dense vertex-level spatial localization. The best spatial baseline still operates over coarse region labels; predicting per-vertex impact on the SMPL-X mesh remains open.
• Brief and repeated impacts. Temporal models degrade sharply on 1–2 frame moments and on queries with multiple occurrences, which the paper identifies as a key bottleneck but does not solve.
• Body-region imbalance. Performance concentrates on the head, back, and buttocks; rebalancing the training label distribution narrows but does not remove the disparity.
• Domain extension. The authors suggest the annotation pipeline could be adapted to other video domains that need human-centered physical event annotations.

Target Audience

Researchers and practitioners in computer vision and multimodal video-language modeling, especially those working on video moment retrieval, spatio-temporal grounding, and human-scene contact estimation. It is also relevant to physical AI and embodied reasoning groups, and to HCI and haptics researchers interested in automatic tactile authoring from video.

Authors’ abstract

Physical reasoning about humans in videos has largely focused on everyday interactions, where contact, pressure, pose, and motion support embodied perception and contact-aware applications. Still, short and forceful properties like impact events remain relatively underexplored, despite their relevance to broader understanding of how abrupt physical events affect the human body. To fill this gap, we introduce human-centric impact localization, a new task that localizes target physical impacts temporally within a video and spatially on the human body. We further present HAPACT, a HumAn-centric imPACT dataset and benchmark built from movie clips, containing over 2.8K video-query pairs with more than 20K fine-grained human-annotated frames. We built HAPACT around viewer-grounded annotation, text-conditioned localization, and supervision at frame-level temporal and SMPL-X-based spatial granularity. To demonstrate the effectiveness of our benchmark, we conduct extensive evaluations of existing methods, employing video moment retrieval models for temporal localization and human-scene contact models for spatial localization. For the spatial task, we propose a text-conditioned baseline by reformulating localization as query-guided body-region selection. Benchmark results show that physical impact localization remains challenging for existing methods, motivating HAPACT as a dedicated benchmark to support physical impact reasoning in videos.