HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics

AI summary

Overview

Research area: Computer vision, specifically egocentric video understanding, hand–object interaction (HOI) recognition, video question answering (Video QA), and multimodal large language models (MLLMs).

Technical level: Advanced. The paper assumes familiarity with video-language architectures, referring/reasoning video object segmentation, contrastive training, and benchmark design.

Scope: This paper introduces HanDyVQA, a human-verified video QA benchmark built on Ego4D that tests fine-grained spatio-temporal reasoning about both the manipulation side and the effect side of hand–object interaction, plus accompanying segmentation masks and an empirical study of how current video foundation models fail on it.

What This Paper Is About

Existing semantic HOI benchmarks evaluate only one side of the interaction at a coarse level — either the manipulation (actions, objects) or the effect (object state changes) — so they do not test fine-grained spatio-temporal reasoning about how a manipulation produces an effect on an object. The authors build HanDyVQA to cover both aspects jointly, with six question types, 11.1K multiple-choice QA pairs, and 10.3K segmentation masks, then benchmark current video-language models against human performance. Their goal is to expose where today's models fall short and to test whether explicit hand/object cues can close part of the gap.

Key Contributions

1. A new benchmark for HOI dynamics. HanDyVQA is a six-category video QA benchmark (Action, Process, Objects, Location, State Change, Object Parts) totaling 11.1K multiple-choice QA pairs drawn from short Ego4D clips, designed to evaluate the progression from manipulation to effect.

2. Segmentation annotations for object- and part-level reasoning. 10.3K masks are provided for the Objects and Object Parts categories, enabling a Reasoning Video Object Segmentation (ReasoningVOS) evaluation where the target must be inferred implicitly from the question rather than from explicit textual grounding.

3. An in-depth failure analysis of current video-language models. The authors quantify zero-shot MCQ performance and ReasoningVOS performance across open-source and proprietary models, and statistically categorize errors into Class/Side Confusion, Interaction/Spatial Relationship Error, Motion Error, and Structural Change Error.

4. A baseline study on explicitly injecting HOI cues. Fine-tuning InternVideo2-Stage2 with 3D hand pose, manipulated-object bounding box tracklets, and object features shows each cue reduces different error types, motivating hand–object-aware video encoders.

Main Findings

• Even the strongest model trails humans by a wide margin. Gemini-2.5-Pro achieved the best average MCQ score at 72.6% (the abstract cites roughly 73%), followed by Qwen2.5-VL-72B at 69.9% and GPT-4o (vision) at 58.9%. Human annotators averaged 96.6% (over 95% in every category; the abstract cites 97%). The paper describes top-1 MCQ performance as at most 68–79% for the best model.

• Object Parts is the hardest category. Gemini-2.5-Pro scored 69.3% on Object Parts and Qwen2.5-VL-72B scored 62.5%, both below their scores on Action (79.1% and 78.0% respectively), indicating component-level grounded reasoning remains difficult.

• Text-only input is above chance but insufficient. GPT-4o (text) reached 41.3% average, above the random baseline of 19.5%, suggesting minor textual bias rather than a solvable shortcut.

• Domain adaptation helps. Among 7B-scale models, LLaVA-Video-7B, fine-tuned on Ego4D, achieved the highest average at 54.8%; the Egocentric-trained dual encoder LaViLa (42.6% avg) outperformed InternVideo2-Stage2 (33.3% avg) and some LLM-integrated models, especially on Action (61.6%) and Objects (68.5%).

• More frames and higher resolution help, but unevenly. Increasing input frames from 1 (~0.2 fps) to 64 (~12.8 fps) and increasing resolution improved performance. Gemini-2.5-Pro showed the most consistent gains from frame count and reached its lowest error rates at 32 frames. mPLUG-Owl3-8B saturated beyond 8 frames in all categories except Objects, which the authors link to its training on eight-frame clips. Motion errors showed the most limited improvement across all models.

• Spatial and interaction errors dominate. The Interaction/Spatial Relationship error class had the highest error rate across models and accounts for the lower performance on Location and Object Parts. Qualitative failures include confusing nearby objects or hand sides, missing hand/object motion, and missing structural or state changes.

• ReasoningVOS scores are far below prior VOS benchmarks. Sa2VA-8B with video input and the question as prompt reached Objects J of 32.0 and Parts J of 10.9, compared to the paper's cited figure of 40+ J for VideoLISA on prior ReasoningVOS tasks. Frame-wise processing sometimes beat video input for small masks (Objects J 36.2, F 29.6 for Sa2VA-8B image input), while video input helped for large masks. Models often over-segment entire objects instead of the queried part.

• Explicit HOI cues improve fine-tuned performance. Fine-tuning InternVideo2-Stage2 on the 1.1K-question training split raised average accuracy from 33.3% (zero-shot RGB) to 51.0% (RGB fine-tuned). Combining RGB with hand pose and object features reached 53.9% average; RGB + Hand + ObjBBox reached 53.5%. Hand pose and bounding box tracklets mainly reduced Class/Side and Interaction/Spatial Relationship errors, while object features mainly reduced Structural Change errors.

• Hand/object-aware pretrained models show mixed results. HelpingHands-L improved Objects (69.2 with 4 frames vs. LaViLa-L's 67.5) but degraded average performance (40.3 vs. 41.8). EgoHOD-L at 16 frames improved Objects (+6.8), Location (+1.1), State (+3.4) over LaViLa-L, with a roughly flat average (42.7 vs. 42.6), but its Motion Adapter did not effectively mitigate motion-related errors.

• GPT-4o abstains on harder questions. GPT-4o text and vision provided valid answers to around 87% and 79% of total questions. Re-evaluating on the 6,629 questions (70.3% of the original 9,431) answered by both variants raised most models' scores; human performance stayed within 0.5 points across categories.

Methodology in Plain English

The team started from Ego4D, choosing it because it contains unscripted, realistic hand–object interactions across many recording locations. They used narration annotations with timestamps and prompted LLMs to judge whether an object was being manipulated in each action, then sampled 2,000 narrations per question category that contained relevant verbs or context. Each item uses a 5-second video segment centered on the narration timestamp, spanning 2.5 seconds before and after.

Questions were drafted by LLMs from templates — for example, "How does the person [verb] [object]?" for Process and "What part of [object] is [effect]?" for Object Parts — with verbs and objects filled in from the narration. Human annotators then verified validity, wrote the correct answer (listing all plausible objects in the Objects category), and refined LLM-generated distractors by removing overlaps, increasing plausibility, and adding harder alternatives. For Objects and Object Parts, annotators sampled about three representative frames per video and drew masks.

For evaluation, the authors selected a mix of dual-encoder video-language models and LLM-integrated models, sampling 16 frames per video and reporting top-1 accuracy for all categories except Objects, where Average Precision is used because of multiple correct answers. For ReasoningVOS they measured Jaccard Index and Boundary F-measure per frame, grouped results by mask size (S/M/L), and tested both question prompts and ground-truth option prompts, plus video versus frame-wise image inputs.

Finally, they fine-tuned InternVideo2-Stage2 on the 1.1K-question training split with separate small encoders (frame-wise MLP plus LSTM) for 3D hand pose (63-dimensional, from WiLoR), manipulated-object bounding box tracklets (4-dimensional, from AMEGO), and object features (768-dimensional, from CLIP), concatenating these with the frozen video feature under a contrastive loss.

Why This Matters

Research impact. HanDyVQA reframes HOI evaluation as a dynamics problem rather than a static recognition problem, and its error taxonomy gives a concrete diagnostic vocabulary (class/side, spatial relationship, motion, structural change) that other benchmark authors can reuse. The finding that motion-related errors barely improve with more frames or resolution is a direct critique of frame-based ViT backbones and points toward architectures that model local entities over time.

Real-world applications:

• Worker assistance systems that need to recognize exactly how a tool is being manipulated and what effect it is producing.
• Dexterous manipulation in robots, where a policy must understand manipulation style and resulting object state.
• Motor function analysis, where fine-grained hand motion and part-level object changes are diagnostic signals.
• Egocentric video assistants and AR/XR guidance that must reason about part-level object states in cluttered real scenes.

Industry relevance. The benchmark shows that scaling general-purpose MLLMs alone leaves a large gap on HOI (best model 72.6% vs. human 96.6%), which matters for companies building embodied AI, robotics, or industrial inspection products where errors in hand-side or part-level localization are costly. It also provides a training/evaluation resource (1.1K train, 0.5K validation, 9.4K test questions) for teams wanting to instruction-tune or adapt models toward manipulation understanding.

Future Directions

• Better integration of HOI cues. The authors note that performance sometimes improved more with fewer cues under their simple concatenation design, and explicitly leave better modeling and integration strategies for future work.

• Architectures that model motion and local entities. The paper argues future models should exploit richer spatio-temporal inputs and explicitly model local hand–object dynamics instead of relying on a fixed ViT backbone pretrained on low-resolution, low-frame-rate video (e.g., 8 fps at 224×224).

• Part-level geometric and grounded reasoning. ReasoningVOS results show models over-segment whole objects rather than queried parts and often miss multiple manipulated objects; improving component-level grounding and spatial relationship reasoning is left open.

• Improving prompt and input strategies for ReasoningVOS. The authors observe that ground-truth option text can be insufficient to describe precise regions in fine-grained HOIs and that frame-wise input sometimes beats video input for small masks, suggesting open questions in how prompts and temporal context should be supplied.

Target Audience

Researchers and engineers working on video-language models, egocentric vision, hand–object interaction, and embodied AI; benchmark and dataset designers who need human-verified, category-balanced evaluation data; robotics and worker-assistance practitioners interested in what current models can and cannot perceive about hands, objects, and their state changes; and anyone studying video object segmentation under implicit, reasoning-based queries.

Authors’ abstract

Hand-object interaction (HOI) inherently involves dynamics where human manipulations produce distinct spatio-temporal effects on objects. However, existing semantic HOI benchmarks focused either on manipulation or on the resulting effects at a coarse level, lacking fine-grained spatio-temporal reasoning to capture the underlying dynamics in HOI. We introduce HanDyVQA, a fine-grained video question-answering benchmark that comprehensively covers both the manipulation and effect aspects of HOI. HanDyVQA comprises six complementary question types (Action, Process, Objects, Location, State Change, and Object Parts), totalling 11.1K multiple-choice QA pairs. Collected QA pairs recognizing manipulation styles, hand/object motions, and part-level state changes. HanDyVQA also includes 10.3K segmentation masks for Objects and Object Parts questions, enabling the evaluation of object/part-level reasoning in video object segmentation. We evaluated recent video foundation models on our benchmark and found that even the best-performing model, Gemini-2.5-Pro, reached only 73% average accuracy, which is far from human performance (97%). Further analysis shows the remaining challenges in spatial relationship, motion, and part-level geometric understanding. We also found that integrating explicit HOI-related cues into visual features improves performance, offering insights for developing future models with a deeper understanding of HOI dynamics.