IntentQA: Intent Question Answering in Videos by Cognitive Context Reasoning

AI summary

Overview

• Research area: Computer vision / video question answering (VideoQA), specifically intent understanding and social intelligence, with cross-modal reasoning that combines vision models, captions and large language models (LLMs).
• Technical level: Advanced. The paper is an IEEE TPAMI journal article building on a prior ICCV 2023 conference paper, and assumes familiarity with VideoQA architectures, contrastive/triplet learning, cross-modal attention, and LLM prompting.
• Scope (one sentence): The paper introduces the IntentQA task and dataset for answering questions about characters' intentions in videos, proposes five automatically generated contrast sets plus a "Contrast Performance Decline" metric for robustness, and presents X-CaVIR, a framework that fuses situational, contrastive and commonsense context through a transparent LLM pipeline.

Note on completeness: the supplied content is truncated inside Section V-A.1 (the first table of ablation results). Dataset statistics and the full model description are present, but the paper's reported accuracy scores, ablation numbers and comparison tables are not included in the available content, so no result values are quoted below.

What This Paper Is About

Most VideoQA systems recognize observable visual facts, but cannot infer the unobservable mental states that drive human behavior, which the authors call the "dark matter" of social intelligence. The same action, such as pointing at a cup, can mean "give me water" in a dining context or "clean the cup" beside a sink of dirty dishes, so intent cannot be read off the pixels alone. The paper's goal is to define an intent-reasoning VideoQA task, supply a large-scale dataset and robustness benchmark for it, and build a model that uses situational, contrastive and commonsense context to answer such questions while making its reasoning traceable.

Key Contributions

1. A new task, IntentQA. The authors define intent question answering in videos, where a model must select an answer from an answer set based on the psychological goals and intentions of characters, rather than on surface facts.
2. A large-scale annotated dataset and a robustness benchmark. The main set contains 4,303 videos and 16,297 QA pairs, drawn from NExT-QA and restricted to inference-style QA types. From this, five contrast sets (gender, verb phrase high-similarity, verb phrase low-similarity, noun phrase high-similarity, noun phrase low-similarity) are generated, along with a new "Contrast Performance Decline" metric.
3. The X-CaVIR framework (eXplainable Context-aware Video Intent Reasoning), which combines three forms of cognitive context: Situational Context via a cross-modal Video Query Language (VQL) module, Contrastive Context via a contrastive learning module, and Commonsense Context via a commonsense reasoning module.
4. A transparent LLM integration. Instead of a black-box merge, the pipeline embeds the visual model's matching scores as confidence values in the prompt alongside dense video captions, so the LLM acts as an explicit reasoner that must output an Option ID, the Option itself, and a "Reasoning Process."

Main Findings

The available content describes the design and announced findings, but stops before the results tables, so numeric outcomes are not reported here.

• Standard accuracy can overestimate ability: The authors state that conventional metrics may overestimate model capability due to dataset biases, citing prior work showing that popular models drop substantially on contrast sets built with subtle perturbations.
• Three context types are each argued to be necessary: Situational Context disambiguates an action from its immediate environment; Contrastive Context sharpens discrimination by comparing the same action across different scenarios; Commonsense Context supplies culturally ingrained knowledge, such as placing a cup on a table in a library meaning "reserving a seat."
• Dataset composition: The main set has 4,303 videos, 16,297 QA pairs, 624 distinct actions, 193 lemmatized verbs and 162 action IDs. QA pairs split into Causal Why (9,424), Causal How (2,633), and Temporal Previous & Temporal Next (4,240), and into training (12,119), validation (2,044) and testing (2,134) QAs at roughly a 6:1:1 ratio.
• Generalization safeguards in the split: Every lemmatized verb appearing in validation or testing is guaranteed to appear at least twice in training, and each video is assigned to only one split to prevent information leakage.
• Confidence Prompting replaces Late Fusion: The preliminary conference model used Late Fusion, a linear combination of a blind (text-only) LLM distribution and visual matching scores. In this version the visual matching scores are embedded directly into the prompt as confidences, and this is claimed to improve performance while making the reasoning explicit.
• Contrast sets were upgraded with LLMs: The authors state that they replaced an earlier template-based substitution pipeline with LLM-driven generation, and extended the replacement criteria to gender, verb phrases and noun phrases, generating 20 candidates per masked span, then selecting the top 15 percent and bottom 15 percent by semantic similarity to form the high-similarity and low-similarity sets.
• Component claims: The paper states that extensive experiments demonstrate the effectiveness of the components, the superiority of X-CaVIR over state-of-the-art baselines, and its stability against perturbations on the contrast sets. The supporting figures are not present in the supplied excerpt.

Methodology in Plain English

The researchers started from NExT-QA, an existing VideoQA dataset of daily social activities, and kept only the inference-style QA types (the Causal and Temporal categories), discarding the factoid Descriptive ones. They parsed each question and answer to identify the key action, lemmatized its verb, and clustered synonyms into action IDs, so that "the same action" could be tracked systematically. Crowd annotators on Amazon Mechanical Turk then filtered candidates against four criteria, including that the key action be physical and observable and that "same actions" really be semantically and physically similar rather than merely similarly worded. At least three annotators cross-validated each sample, and only unanimous samples were kept. Because the dataset is deliberately constructed so that the same action under different contexts yields different intents, the same pointing gesture appears with different correct answers in different videos.

For robustness testing, they reused an existing strategy for auto-generating text-based contrast sets for video-language tasks but replaced its simple template substitutions with LLM generation. Gender contrasts use a fixed mapping table (for example, "he" to "she", "man" to "woman", "boy" to "girl"), applied only when the correct answer contains a gender-sensitive term. Verb and noun contrasts mask the pivotal verb or noun phrase identified by dependency parsing, prompt InstructGPT for 20 replacements, and then split them by semantic similarity: the closest 15 percent become high-similarity distractors that demand fine-grained visual discrimination, and the furthest 15 percent become low-similarity distractors that are easier to reject. One option from each subset is inserted back into the original answer set.

The model itself builds on a simplified version of VGT as its backbone. Frame features and region features are extracted from the video, the region features pass through N DGT modules to form a region graph, and this is concatenated with the frame features. Questions and answers are concatenated and encoded with BERT. The VQL module then lets the region graph query the language features: it computes a similarity matrix between the region graph and the language features, uses that matrix to project the language features into the visual feature space, and adds the result back to the region graph to produce a question-relevant cross-modal graph, which is the Situational Context. A multi-head self-attention transformer fuses the frame/region graph with this cross-modal graph.

For Contrastive Context, each training QA (the anchor) is paired with a positive and a negative example chosen by two similarity conditions. Action similarity is controlled by action consistency, lemmatized-verb consistency, or action ID consistency; whether two samples with the same action are positive or negative depends on the WUPS similarity between their correct answers, compared against thresholds t1 (positive) and t2 (negative). The top-k nodes most relevant to the question are taken from each of the three samples' cross-modal graphs, the positive and negative features are aligned to the anchor, and a triplet margin loss pulls the anchor toward the positive and away from the negative. The total training objective sums three cross-entropy losses (for the anchor, positive and negative) and the triplet loss.

At inference, the Confidence Prompting pipeline formats the answer set and the visual model's matching scores into candidate options annotated with confidence, and optionally adds dense captions of the video frames produced by Qwen 7B. This prompt, along with the question, goes to the LLM, which must return the option ID, the option text and a reasoning process. A response counts as correct only if the option ID correctly matches the option text, which the authors say is to avoid overestimating the LLM in cases where it guesses correctly by chance.

Why This Matters

Impact on research. The paper pushes VideoQA from recognizing what happened toward reasoning about why it happened, and argues it is the first VideoQA work focused on intent. It also treats evaluation as a first-class problem: by publishing five contrast sets and a "Contrast Performance Decline" metric, it offers a template for testing whether a model genuinely understands intents or merely exploits dataset biases. The explicit reasoning output also provides an interpretability handle that the authors say was useful for diagnosing qualitative failure cases.

Real-world applications (as motivated by the paper's framing of social intelligence and non-verbal cues):

• Assistive and companion systems that need to infer what a person wants from a gesture, such as pointing at a cup or a screen.
• Human-robot interaction, where a robot must infer a person's goal from an observed action before acting, a setting the related work section connects to trajectory prediction and hierarchical intent inference from RGBD video.
• Human behavior and interaction analysis, where interpreting goal-directed action at scale supports studying social activities.
• Robust multimodal assistants and benchmarking pipelines, where confidence-aware LLM prompting and contrast-set testing help catch models that look accurate but rely on shortcuts.

Industry relevance. The system-based design, which keeps a visual model and an LLM as separate components coordinated by a prompt, is a practical alternative to end-to-end visual-language models. The authors argue that integrated LLM-plus-visual-encoder models do not necessarily beat system-based methods, partly because open-source LLM backbones are weaker than proprietary ones, and that integrated models are hard to debug because it is unclear what information the visual component contributes. Their pipeline instead exposes both the visual confidences and the LLM's stated reasoning, which is attractive for auditing deployed systems. The paper also reports comparing ChatGPT and GPT-4 as LLM backbones.

Future Directions

• Closing the robustness gap. The paper introduces the Contrast Performance Decline metric but does not, in the available content, specify how large the decline is or which contrast type hurts most; establishing which perturbations remain hardest is the natural next step.
• Reducing dependence on proprietary LLMs. The authors note that system-based methods' performance is limited by weaker open-source backbones relative to GPT-4 and ChatGPT, implying value in closing that gap without losing interpretability.
• Improving the caption-reasoning interface. Captions come from a single model (Qwen 7B); whether better or differently structured visual descriptions change intent inference is left open.
• Broadening and diagnosing intent coverage. The dataset centers on 624 actions, 193 lemmatized verbs and 162 action IDs from NExT-QA's daily social activities, so extending to new action vocabularies, cultures and domains, and using the LLM's stated reasoning to systematically categorize failure cases, are open directions.

Target Audience

Researchers and graduate students in computer vision, video-language understanding and multimodal reasoning who work on VideoQA, visual reasoning or social intelligence. It is also relevant to practitioners building multimodal assistants, LLM-plus-tools pipelines, or human-robot systems that must infer intent from video, and to those focused on benchmark design and robustness evaluation. Readers without background in cross-modal architectures, contrastive learning and prompting would likely find the method sections demanding.

Authors’ abstract

Video understanding requires intelligent agents to transcend mere recognition of visual facts and comprehend the underlying intents behind human actions (often termed the "dark matter" of social intelligence). To bridge the gap between visual observation and intent reasoning, we introduce a novel task, IntentQA, and contribute a large-scale VideoQA dataset specifically tailored for this purpose. However, recognizing that standard metrics may overestimate capabilities due to dataset biases, we go beyond simple accuracy to rigorously evaluate model robustness. We augment the benchmark by generating five distinct contrast sets via Large Language Models (LLMs) and introducing a "Contrast Performance Decline" metric. We propose the X-CaVIR (eXplainable Context-aware Video Intent Reasoning) framework, which leverages three types of "Cognitive Context" to enhance video analysis: i) Situational Context via a cross-modal Video Query Language (VQL) module, ii) Contrastive Context via a Contrastive Learning module, and iii) Commonsense Context via a Commonsense Reasoning module. Crucially, to overcome the opacity of traditional black-box models, we refine the integration of LLMs within X-CaVIR by employing a transparent pipeline that synergizes video captions with VQA model outputs. This approach not only improves performance by effectively utilizing rich commonsense knowledge but also renders the reasoning process explicitly interpretable. Extensive experiments demonstrate the effectiveness of our components, the superiority of X-CaVIR over state-of-the-art baselines, and its stability against perturbations on the contrast sets.