Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs

AI summary

Overview

• Research area: Mechanistic interpretability of video large language models (VideoLLMs) for video question answering (VideoQA).
• Technical level: Intermediate. Readers benefit from familiarity with transformer attention, multimodal LLMs, and evaluation benchmarks, though the core narrative is presented conceptually.
• Scope: The paper reverse-engineers the internal information flow of VideoLLMs to identify which attention pathways carry temporal reasoning information and which can be pruned without harming VideoQA accuracy.

What This Paper Is About

VideoLLMs answer questions about videos by combining a vision encoder with a language model, but almost all prior work has focused on external design choices such as scaling instruction data, selecting key frames, or compressing video tokens. The internal question of where and how these models extract temporal information from video tokens and propagate it into text tokens to produce an answer has remained largely unexplored. This paper uses mechanistic interpretability tools to trace that flow layer by layer and reveals a consistent, sparse set of information pathways that are sufficient for accurate VideoQA.

Key Contributions

1. A layer-wise blueprint of temporal reasoning in VideoLLMs, showing that temporal reasoning begins with cross-frame interactions within video tokens in early-to-middle layers, followed by video-language integration on temporal keywords in middle layers, and answer generation in middle-to-late layers.
2. Evidence that VideoQA instruction tuning, not image pretraining, is what induces cross-frame dependency: disrupting cross-frame attention harms LLaVA-NeXT-7B-Video-FT across all five analyzed tasks while the image-only LLaVA-NeXT-7B remains mostly unaffected.
3. Identification of temporal keywords (including multiple-choice options) as information integration checkpoints, with task-dependent routing between a direct video-to-option route and an indirect video-to-question-to-option route.
4. A demonstration that retaining only the identified effective pathways preserves VideoQA accuracy while suppressing a large fraction of attention edges, for example retaining only 42% of edges in LLaVA-NeXT-7B-Video-FT and 58% in VideoLLaMA3-7B.

Main Findings

• Cross-frame interaction is the starting point: Blocking cross-frame attention in early-to-middle layers degrades prediction probability for LLaVA-NeXT-7B-Video-FT across all five TVBench tasks, whereas LLaVA-NeXT-7B shows little layer-wise sensitivity in most scenarios.
• Intervening on the first half of layers causes large accuracy drops: Blocking cross-frame attention in layers 1–16 produced accuracy drops of −24.1% (Action Antonym), −20.2% (Action Sequence), −18.0% (Scene Transition), −44.8% (Moving Direction), and −60.8% (Object Count), with open-ended responses becoming incorrect or even opposite to the video content.
• Spatial concepts emerge before temporal concepts: Logit Lens applied to LLaVA-NeXT-13B-Video-FT on Action Sequence videos shows spatial concepts appearing in very early layers and temporal concepts developing from the middle layers onward, with temporal concepts occupying remaining token positions rather than displacing stabilized spatial ones.
• Cross-frame interaction enables video-text alignment: With normal cross-frame attention, question tokens such as "begins" and "ends" attend to semantically relevant video regions; when cross-frame attention is blocked, alignment fails and text tokens attend to positionally proximate regions instead.
• Video information flows through option tokens: The last token derives little direct flow from non-option question tokens, while information from the true option propagates in middle-to-late layers. In Action Antonym, Action Sequence, and Scene Transition the direct video-to-true-option route dominates, whereas in Moving Direction and Object Count information is first routed to the non-option question and then to the true option.
• Answer readiness begins around the 20th layer: Layer-wise probing of the last token shows the probability for the true option rising abruptly starting around the 20th layer, coinciding with completion of the video-to-question flow, and the true option rapidly dominates rather than competing gradually with other options.
• Effective pathways suffice for VideoQA: On TVBench and TOMATO, restricting attention to effective pathways gives LLaVA-NeXT-7B-Video-FT 51.2 and 29.2 (versus 51.5 and 30.2 with full causal attention) using 10.8M of 25.7M attention edges; LLaVA-NeXT-13B-Video-FT gives 54.6 and 27.4 (versus 55.1 and 27.2) using 14.3M of 32.2M; Mini-InternVL-4B-Video-FT gives 56.0 and 31.2 (versus 56.0 and 32.2) using 29.6M of 74.6M; VideoLLaMA3-7B gives 57.2 and 28.7 (versus 55.2 and 28.0) using 11.4M of 19.9M. Random blocking of the same proportion of edges collapses performance, for example to 22.2 and 13.9 for VideoLLaMA3-7B.
• Failures originate in early spatiotemporal representation: In failed Action Antonym samples, cross-modal integration patterns resemble those of successful samples, and two failure modes appear when cross-frame interaction is disabled: one where the incorrect option's probability decreases (spurious cross-frame attention) and one where it increases (reliance on static bias).

Methodology in Plain English

The researchers take a model that started as an image-text MLLM (LLaVA-NeXT-7B) and fine-tune it with VideoChat2-IT for 3 epochs to create LLaVA-NeXT-7B-Video-FT, their primary subject. They also analyze LLaVA-NeXT-13B-Video-FT, Mini-InternVL-4B-Video-FT, and VideoLLaMA3-7B. Training and inference use 8-frame sampling with 144 tokens per frame.

For analysis they use five TVBench tasks covering different reasoning types: Action Antonym, Action Sequence, Scene Transition, Moving Direction, and Object Count. They restrict analysis to instances the model answers correctly, to avoid noise from guesses.

Two interpretability techniques drive the work. Attention Knockout sets the attention mask at a given source-target position to negative infinity, disconnecting a specific attention edge during inference; the researchers then measure the percentage relative change in the probability of the model's original answer. When testing cross-frame dependencies they block vision tokens from attending to tokens in preceding frames within a window of 9 layers centered on each layer. Logit Lens projects hidden states of video tokens at every layer through the language model head, turning them into vocabulary logits so the emergence of spatial and temporal keywords can be counted and visualized across layers. Together these let the authors trace video-to-question, question-to-last-token, and video-to-last-token flows layer by layer, then test whether keeping only the identified critical ranges (for example cross-frame interactions at L6–15, video-to-question at L6–20, question-to-last at L16–25, with video-to-last and last-to-last disabled) preserves benchmark accuracy.

Why This Matters

Impact on research. The paper provides a first blueprint of temporal reasoning inside VideoLLMs, showing which attention pathways matter and which are expendable. It extends mechanistic interpretability from image-based MLLMs to spatiotemporal inputs, and its results suggest that findings from image-text models do not automatically transfer once video instruction tuning is applied.

Real-world applications.

• Inference efficiency: nearly half to more than half of attention edges can be removed while retaining accuracy, which points toward cheaper video model serving.
• Model debugging: the failure analysis distinguishes errors caused by spurious cross-frame attention from errors caused by static-scene bias, giving a diagnostic for video model development.
• Training design: knowing that temporal concepts should form in middle layers and video-text checkpoints occur around option or verb tokens can inform training objectives and data construction.
• Architecture research: the effective layer ranges support early-exit designs that truncate computation once video-language integration completes.

Industry relevance. Companies deploying video understanding systems face heavy compute costs from long token sequences. A validated recipe for pruning attention edges and for truncating late-layer computation directly reduces inference overhead, and the observation that temporal concepts form later than spatial concepts offers a concrete target for improving temporal robustness in production models.

Future Directions

1. Pathway regularization during training. The authors suggest that suppressing dominant information pathways could push models to explore alternatives and use more of their representational capacity.
2. Reducing static-scene bias. The failure analysis indicates that some errors come from defaulting to uninformative static cues, implying that training objectives or architectural constraints promoting early formation of visual representations could improve temporal reasoning.
3. Early-exit strategies. Because token interactions beyond the effective ranges contribute marginally to final decisions, adaptively truncating redundant computation could cut inference overhead while preserving accuracy.
4. Extension to broader settings. The main analysis covers multiple-choice VideoQA, with open-ended QA, long-form video understanding, and spatial reasoning addressed in the Appendix, leaving full generalization of the blueprint to other formats and architectures as further work.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, model interpretability, and efficient inference benefit most. The paper is also useful for practitioners who need to justify pruning or early-exit decisions in deployed video models, and for students entering mechanistic interpretability who want a worked example of applying Attention Knockout and Logit Lens to a spatiotemporal model.

Authors’ abstract

Video Large Language Models (VideoLLMs) extend the capabilities of vision-language models to spatiotemporal inputs, enabling tasks such as video question answering (VideoQA). Despite recent advances in VideoLLMs, their internal mechanisms on where and how they extract and propagate video and textual information remain less explored. In this study, we investigate the internal information flow of VideoLLMs using mechanistic interpretability techniques. Our analysis reveals consistent patterns across diverse VideoQA tasks: (1) temporal reasoning in VideoLLMs initiates with active cross-frame interactions in early-to-middle layers, (2) followed by progressive video-language integration in middle layers. This is facilitated by alignment between video representations and linguistic embeddings containing temporal concepts. (3) Upon completion of this integration, the model is ready to generate correct answers in middle-to-late layers. (4) Based on our analysis, we show that VideoLLMs retain their VideoQA performance by selecting these effective information pathways while suppressing a substantial amount of attention edges, e.g., 58% in LLaVA-NeXT-7B-Video-FT. These findings provide a blueprint for how VideoLLMs perform temporal reasoning and offer practical insights for improving model interpretability and downstream generalization. Our project page with the source code is available at https://map-the-flow.github.io