From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making

AI summary

Overview

Research area: Mechanistic interpretability of Vision-Language Models (VLMs), specifically causal information flow between video and text in multimodal decision-making.

Technical level: Advanced. The paper assumes familiarity with transformer self-attention, attention masks, logit-lens readout, and activation-level interventions.

Scope: A layer-wise causal intervention study of how video information reaches language-based decisions in two 7B VLMs across spatial, causal, and temporal reasoning subsets.

What This Paper Is About

VLMs are usually judged only by their final answers, but a correct answer does not show whether the model actually used the visual input or just exploited linguistic regularities in the prompt. This paper asks where and how visual information from a video enters the textual representations a model uses when choosing between two candidate descriptions, and whether that flow differs across spatial, causal, and temporal reasoning. The authors answer by blocking specific video-to-text attention connections and measuring the effect on the plausibility of each candidate.

Key Contributions

1. A causal intervention framework for tracing video-to-text information flow in VLMs during caption–foil decision-making, implemented by modifying pre-softmax attention-score matrices directly in the forward pass rather than through hook-based approximations.
2. An analysis of which prompt regions (question, option A content, option B content, assistant content, vision) and which token types (nouns, verbs, prepositions, category-critical tokens) mediate visual grounding.
3. An investigation of how these dynamics vary across spatial, causal, and temporal relations, including a distinct pattern for temporal items.
4. A release of code and data on GitHub.

Main Findings

• Answer options are the grounding site: Visual information is integrated mainly when the model processes the two candidate answer options, not the task instruction or the assistant response. Masking vision-to-option attention produces the highest flip rates and largest accuracy changes; masking the question and assistant regions produces lower flip rates and smaller changes.
• No direct video-to-assistant pathway: Blocking vision-to-assistant attention produces only minor differences relative to the unmasked none condition, suggesting visual information survives through visually enriched textual carriers rather than being routed directly to the final decision.
• Late-layer effect for spatial and causal: In early and middle layers, values are close to zero or biased toward the incorrect foil (for example around layers 4–6 for the Causal subset in LLaVA-OneVision and around layer 4 in Qwen2.5-VL); from layer 22 onward the trend reverses and both models increasingly prefer the correct caption.
• Nouns act as semantic anchors: Across models and subsets, nouns are consistently among the most affected categories under vision-to-token knockout, for both caption and foil. LLaVA-OneVision shows a more localized pattern, with the largest drops often occurring for category-critical tokens, especially for captions.
• Verbs matter more for temporal items: In the Temporal subset, verbs show a stronger dependence on the visual input than in Spatial and Causal subsets, consistent with the need to process visually grounded event information rather than only grounded entities.
• Model differences: Qwen2.5-VL shows the strongest drop when video access is masked for specific linguistic elements within the options, suggesting it relies more strongly on visual information during answer processing. LLaVA-OneVision is less affected overall, with the exception of a consistent effect around layer 25 across all subsets.
• Temporal items are fragile: Temporal caption–foil decisions appear to be determined later in the network. The averaged preference for the foil across items is mainly driven by incorrect cases: both models initially favor the caption, and the foil becomes prominent only in late layers.
• Anomalous temporal-layer effect: For LLaVA-OneVision, an intervention effect appears around layers 20–21 when masking the question region, and at layer 23 the vision-to-assistant knockout has a positive effect, assigning higher probability to the caption, contrary to the other masking regions and to the none condition.
• Temporal performance is below chance under full-sentence scoring: Full-sentence scoring gives comparable results to A/B-label scoring on Spatial and Causal but substantially lower accuracy on Temporal (Qwen2.5-VL 0.77 A/B vs. 0.43 caption/foil; LLaVA-OneVision 0.73 vs. 0.44).
• Linguistic bias in temporal markers: In the text-only condition, both models favor captions with Before over After (Qwen 0.82 vs. 0.42; LLaVA 0.75 vs. 0.36). With full visual input, When remains below chance (0.41 for Qwen, 0.28 for LLaVA) and While stays below chance (0.13 for Qwen, 0.17 for LLaVA), with While being the largest marker group.
• Text-only is always worse: The text-only baseline consistently underperforms the full-input none condition across both models and all three subsets, indicating linguistic priors alone are insufficient.
• Data scale: Each MAIA subset comprises 1,600 caption–foil pairs associated with 100 videos, corresponding to 16 pairs per video within each subset and 4,800 pairs overall.

Methodology in Plain English

The authors use a caption–foil task: given a video and a multimodal prompt, the model must pick the more plausible of two sentences, where the foil is a minimally altered version of the caption that introduces a subtle but semantically critical inconsistency. The caption and foil are randomly assigned to options A and B to avoid position bias. The data comes from the Italian MAIA benchmark, using the Spatial, Temporal, and Causal subsets. Both models (Qwen2.5-VL 7B and LLaVA-OneVision 7B) are given each video sampled into 16 frames.

To find out where visual information is used, the authors apply Attention Knockout. They modify the attention computation during the forward pass so that attention from a chosen query token to a chosen span of key tokens is suppressed by setting the corresponding scores to negative infinity, removing their contribution after the softmax. Interventions are defined as (query, key-start, key-end) tuples and applied globally across all transformer layers in a single forward pass. Two directions are possible: vision-to-text (the main analysis, blocking textual queries from attending to visual tokens) and text-to-vision (a control analysis requiring an inverted prompt because causal attention cannot support it in the standard layout).

Masking is applied at two granularities. Region-based masking groups the prompt into the question, option A content, option B content, assistant content, and a vision region, identified automatically through token-level position mappings. Token-level masking decomposes the option regions using part-of-speech annotations from spaCy's it_core_news_sm model, isolating nouns, verbs, prepositions, and category-critical tokens — the minimal lexical elements that distinguish caption from foil. Layer-wise scores are read out with a logit-lens and scored with a forced-target procedure: each candidate sentence is placed in the assistant-response position and its tokens scored autoregressively, with sentence-level log-probability obtained by summing token log-probabilities. Two baselines anchor the analysis: none (standard multimodal forward pass) and text-only (unimodal linguistic-prior baseline). As a robustness check, the analysis was replicated on English translations of 200-item samples drawn from each original MAIA subset.

Why This Matters

Impact on research. The paper shifts evaluation from final accuracy to internal causal mechanisms, addressing the risk of unimodal collapse — where a model reaches the correct answer without genuinely relying on the visual modality. It extends attention-knockout work, which has largely focused on image-based tasks, into video, and connects foiling-style contrastive evaluation with mechanistic interpretability.

Real-world applications:

• Video question answering and video-based assistants, where knowing that answer options are the grounding site can inform how prompts are constructed and how much visual evidence is needed.
• Accessibility tools that describe video content, where models must ground descriptions in what is actually shown rather than in plausible-sounding generic language.
• Media verification and content moderation pipelines that depend on temporal ordering of events (what happened before or after what), where the paper shows current models are fragile.
• Evaluation and red-teaming of multimodal systems, using causal interventions rather than accuracy alone to detect reliance on linguistic shortcuts.

Industry relevance. The finding that temporal decisions are made late in the network and are partly driven by lexical biases associated with specific temporal expressions (While, When, Before) has direct implications for anyone deploying VLMs on video. It suggests that reported benchmark accuracy can overstate genuine temporal understanding, and that reliability auditing should include the linguistic formulation of candidate descriptions.

Future Directions

• Evaluate models beyond the two relatively small 7B VLMs tested, to test how far the observed information-flow patterns generalize across architectures.
• Systematically analyze the differences between A/B-label scoring and full-sentence scoring, since the authors note the choice of scoring affects behavior, particularly for Temporal items.
• Investigate the role of the input language, given that temporal dynamics in the English replication appeared more stable across layers while still differing from Spatial and Causal cases.
• Probe why temporal relations are processed differently from causal and spatial relations, including whether models can be made to recover and maintain sequential information across video frames rather than relying on direct access to the video in late layers.

Target Audience

Researchers and graduate students in multimodal NLP and interpretability; engineers building or auditing video-language systems who need to know where visual evidence actually enters a model's decision; and evaluation specialists concerned with distinguishing genuine visual grounding from linguistic shortcuts. A background in transformer internals, attention mechanisms, and causal interventions is assumed.

Authors’ abstract

Vision-Language Models are commonly evaluated through their final predictions, but understanding whether these decisions are grounded in visual evidence requires tracing how visual information contributes to language-based decisions. With this purpose in mind, we investigate cross-modal information flow in a video-based generative multiple-choice-like setting by applying a layer-wise causal intervention on video-text attention pathways. We target spatial, causal, and temporal visual reasoning. Our results show that visual information is mainly integrated while the model processes the candidate answer options, which serve as the primary textual grounding sites for the final decision. We further show that nouns play an important role as semantic anchors during multimodal enrichment, while verbs are more relevant when temporal relations are processed. Finally, we identify a distinct pattern in temporal reasoning, suggesting that VLMs struggle to reconstruct sequential information across video frames, but we remark that such fragility may also reflect linguistic biases associated with specific temporal expressions used for defining the relation between events within a scene.