Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically video large language models (Video-LLMs) and temporal reasoning in video question answering.

Technical level: Intermediate. The paper assumes familiarity with vision transformers, attention mechanisms, and multimodal LLM architectures, but the core idea is explained clearly enough for readers with basic deep learning background.

Scope: The paper diagnoses temporal reasoning failures in current Video-LLMs on the SSv2-T10 action recognition subset and proposes STAVEQ2, a Qwen2-VL variant with stacked temporal attention inside the vision encoder, evaluated on VITATECS, MVBench, Video-MME, and SSv2.

What This Paper Is About

Current Video-LLMs handle spatially grounded questions reasonably well but fail on questions that depend on how a scene changes over time — for example, distinguishing "pulling something from left to right" from "pulling something from right to left." The authors first show that this failure is architectural rather than a data problem, then add dedicated temporal attention blocks directly inside the vision encoder so that temporal structure is captured before visual tokens reach the language model. Accepted to NeurIPS 2025.

Key Contributions

1. An analysis showing that Video-LLMs such as Qwen2-VL and InternVideo2-Chat struggle with fine-grained temporal reasoning, and that this weakness persists under in-context learning and fine-tuning.
2. STAVEQ2 (Stacked Temporal Attention in Visual Encoders for Qwen2-VL), a Video-LLM architecture that inserts temporal attention modules after spatial attention blocks within the vision encoder. The authors state this is the first work to efficiently include dedicated temporal attention blocks in the vision encoder of Video-LLMs for video question answering.
3. New state-of-the-art results on the SSv2 action recognition benchmark by applying the same temporal attention mechanism to the InternVideo2 video foundation model (vision-only): InternVideo2 1B + STA reaches 78.0%, exceeding InternVideo2 6B by 0.5%.
4. Improvements of up to +5.5% across VITATECS, MVBench, and Video-MME benchmarks.

Main Findings

• Zero-shot temporal failure: On the constructed SSv2-T10 dataset, zero-shot accuracy was 14.87% for Qwen2-VL 2B, 21.91% for Qwen2-VL 7B, 30.60% for InternVideo2-Chat 8B, and 19.38% for LLaVA-NeXT-Video 7B.
• Class list helps but is insufficient: Providing the 10 candidate classes in the prompt raised accuracy to 24.01% (Qwen2-VL 2B), 35.91% (Qwen2-VL 7B), 46.11% (InternVideo2-Chat 8B), and 31.46% (LLaVA-NeXT-Video 7B), yet models still often failed to select the correct class.
• In-context learning degrades performance: Adding 1, 3, or 5 in-context examples lowered accuracy for all models. At 5 examples, scores were 8.92%, 20.72%, 10.41%, and 16.32% respectively. The authors conclude current video-language models lack sufficient in-context learning capability for VQA.
• Fine-tuning is not enough: After fine-tuning InternVideo2-Chat on SSv2-T10, confusion between temporally mirrored actions such as "Pulling [something] from left to right" versus "Pulling [something] from right to left" remained high.
• Temporal attention fixes the confusion: Adding stacked temporal blocks made InternVideo2-Chat able to distinguish these actions. Quantitatively, InternVideo2-Chat 8B went from 84.17% to 95.18% accuracy on SSv2-T10 with STA, a gain of 11.01%.
• New SSv2 state of the art: InternVideo2 1B + STA achieved 78.0% on the full SSv2 dataset, compared to 77.5% for InternVideo2 6B, 77.2% for InternVideo, 77.1% for InternVideo2 1B, 77.3% for MVD-H, 76.7% for MVD-L, 76.8% for VideoMAE V2-H, and 77.0% for VideoMAE V2-g. InternVideo2 1B + STA has about 1.3B parameters.
• Visual similarity matching improves: On the SSv2-VSM subset of 8,471 samples, Qwen2-VL 2B improved from 68.65% to 72.19% and Qwen2-VL 7B improved from 73.15% to 76.05%.
• Benchmark gains: STAVEQ2 2B reached 65.1 MVBench and 56.2 / 61.3 Video-MME (without/with subtitles) versus 63.2 and 55.6 / 60.4 for Qwen2-VL 2B. STAVEQ2 7B reached 70.1 MVBench and 66.8 / 71.8 Video-MME versus 67.0 and 63.3 / 69.0 for Qwen2-VL 7B. STAVEQ2 72B reached 74.5 MVBench and 73.9 / 79.9 Video-MME.
• Comparison to GPT-4o and LLaVA-Video: The paper reports STAVEQ2 outperforming GPT-4o by 2/2.7 and LLaVA-Video 72B by 3.3/3 on Video-MME accuracy. GPT-4o scores 71.9 / 77.2 and LLaVA-Video 72B scores 70.6 / 76.9 on that benchmark.
• VITATECS dominance: STAVEQ2 72B scored 92.8 (compositionality), 90.1 (direction), 82.3 (intensity), 87.9 (localization), 90.3 (sequence), and 92.8 (type), the highest reported in the table across all six aspects.
• Ablation — attention order and head count: On SSv2-T10 with STAVEQ2 2B, spatial-first attention with head scale 0.25 reached 76.04%, versus 73.14% for Qwen2-VL 2B, 71.18% for head scale 0.5, 58.34% for head scale 1.0, and 73.20% for temporal-first at 0.25 scale.
• Ablation — block count and placement: Using 32 STA-enhanced transformer blocks across all layers gave 76.04%, while 16 uniformly distributed blocks gave 74.73% and 16 blocks in the first layers gave 74.97%. The authors note placement matters less than the total number of blocks.
• Data-mix sensitivity for similarity matching: On SSv2-VSM, 50% positive samples yielded 25.52% accuracy, 80% positive yielded a peak of 71.25%, and 91% positive dropped to 49.18%. Removing textual descriptions of reference actions reduced the 80% setting from 71.25% to 68.65%.

Methodology in Plain English

The researchers first built a diagnostic dataset by narrowing Something-Something v2 (SSv2), which has over 220K videos across 174 action classes, down to SSv2-T10: 14,462 videos spanning 10 classes chosen as pairwise temporal opposites (for example, moving something away from something versus closer to something). They tested four Video-LLMs zero-shot, with the class list supplied, and with 1, 3, and 5 in-context video examples.

Their model, STAVEQ2, modifies the Qwen2-VL vision encoder. Each transformer block first performs spatial attention within a frame, then performs temporal attention across frames for each patch position, followed by a feed-forward MLP with residual connections and layer normalization. The temporal module uses up to four times fewer attention heads than spatial attention while keeping the head dimension, and uses 1D rotary position embeddings rather than the 2D version used for spatial encoding. The output projection of the temporal heads is initialized to zero so the encoder starts out behaving like the original model.

Training is two-stage. In the first stage everything is frozen except the temporal attention blocks and their associated layer normalizations, trained with linear warmup. In the second stage, LoRA adapters are added to the linear layers of both the vision encoder (attention projections and MLP) and the LLM, and the temporal blocks and LoRA adapters are trained jointly. Training data comes from a curated subset of WebVid, converted into multi-turn question-answer pairs by prompting the Qwen2 7B LLM, referred to as WebVid-QA. Experiments ran on 64 NVIDIA A100 GPUs. Answers were scored by another LLM, Qwen2-7B, acting as a judge returning "Yes" or "No."

Why This Matters

Impact on research: The paper argues that temporal understanding in Video-LLMs is an encoder-level problem, not just a matter of better LLM reasoning or more training data. It shows that adding temporal attention improves performance even in InternVideo2, which already uses joint spatiotemporal attention, and that the gain is not simply due to extra parameters — a 1B model with STA beats a 6B model on SSv2.

Real-world applications:

• Action recognition and monitoring in video archives or surveillance, where direction of motion matters.
• Robotics and instruction following that depend on the order and direction of manipulation steps.
• Content moderation or compliance review where an action's temporal direction changes its meaning.
• Assistive tools for video search and indexing where users ask questions about how events unfolded.

Industry relevance: The method is parameter-efficient and can be retrofitted onto an existing Video-LLM checkpoint, which matters for teams that cannot pretrain video models from scratch. LoRA-based second-stage training and the reduced head count keep compute requirements modest relative to full fine-tuning, and the authors report the architecture scales to 72B parameters.

Future Directions

• Full pretraining from scratch and scaling beyond 72B parameters, which the authors explicitly list as future work given their resource constraints.
• Extending the evaluation to longer-video settings and benchmarks, since InternVideo2-Chat is restricted to 8 input frames and could not be used on longer-video benchmarks.
• Investigating why in-context learning degrades video action recognition performance and whether the temporal encoder mitigates it — the paper documents the drop but the truncated content does not report a resolution.
• Determining whether strategic placement of temporal blocks matters more at larger scale, since the ablation at 2B showed placement effects smaller than the effect of total block count.

Target Audience

Researchers and engineers working on Video-LLMs, multimodal foundation models, and video understanding benchmarks, particularly those interested in architecture-level changes to vision encoders rather than prompting or scaling strategies. It is also useful for practitioners who want a parameter-efficient way to add temporal capability to an existing Qwen2-VL-based system.

Authors’ abstract

Despite significant advances in Multimodal Large Language Models (MLLMs), understanding complex temporal dynamics in videos remains a major challenge. Our experiments show that current Video Large Language Model (Video-LLM) architectures have critical limitations in temporal understanding, struggling with tasks that require detailed comprehension of action sequences and temporal progression. In this work, we propose a Video-LLM architecture that introduces stacked temporal attention modules directly within the vision encoder. This design incorporates a temporal attention in vision encoder, enabling the model to better capture the progression of actions and the relationships between frames before passing visual tokens to the LLM. Our results show that this approach significantly improves temporal reasoning and outperforms existing models in video question answering tasks, specifically in action recognition. We improve on benchmarks including VITATECS, MVBench, and Video-MME by up to +5.5%. By enhancing the vision encoder with temporal structure, we address a critical gap in video understanding for Video-LLMs. Project page and code are available at: https://alirasekh.github.io/STAVEQ2/.