TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understanding

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically efficient architectures for long-video understanding.

Technical level: Advanced. The paper assumes familiarity with attention, state-space models (Mamba), token compression in vision-language models, and standard video QA benchmarks.

Scope: The paper introduces TimeViper, a hybrid Mamba-Transformer vision-language model with an in-LLM token-transfer module (TransV) that compresses vision tokens into instruction tokens, allowing the model to process videos exceeding 10,000 frames.

What This Paper Is About

Long videos overwhelm today's multimodal models. The authors note that a one-hour video sampled at 1 frame per second, with each frame encoded into 768 vision tokens, produces roughly 2.7 million tokens, exceeding even the million-token context limit of Gemini. Two bottlenecks follow: Transformer attention scales quadratically with context length, and long video sequences contain heavy redundancy.

The goal is a single model that is both efficient enough to ingest hour-long videos and capable enough to answer questions, localize events in time, and produce detailed captions. TimeViper addresses this by combining Mamba-2 layers with a small number of attention layers, and by moving visual information from redundant vision tokens into the text instruction tokens inside the LLM rather than discarding it.

Key Contributions

1. TimeViper, a hybrid Mamba-Transformer MLLM. The LLM backbone contains 27 Mamba-2 layers, 4 self-attention layers, and 25 MLP layers, paired with a ViT visual encoder and a projector that applies token merging (ToMe) so that each frame becomes 16 vision tokens. With in-LLM compression, it processes over 10,000 frames.

2. Discovery of the vision-to-text information aggregation phenomenon. By blocking information flow between token types at different layers, the authors show that visual information progressively flows into instruction tokens as LLM depth increases, leaving vision tokens severely redundant in deep layers.

3. TransV, a token information transfer module. TransV uses gated cross-attention to transfer and compress vision tokens into instruction tokens, rather than dropping tokens outright. A learnable scalar (initialized to zero, normalized via tanh) controls how much information is aggregated. The paper describes TransV as the first token-transfer module that compresses vision tokens into text tokens inside a hybrid LLM.

4. Interpretability analysis of hybrid layers. The paper visualizes attention behavior in both Mamba and self-attention layers, characterizing Mamba layers as exhibiting diverse patterns (sparsity, locality, globality) and identifying attention sinks and declining attention to vision tokens in the self-attention layers.

Main Findings

• Vision tokens become nearly fully redundant with depth. The authors drop vision tokens at different layers using uniform dropping and attention-guided dropping. For the tested tasks, vision tokens are critical in shallow layers but become nearly 100% redundant in deep layers; even discarding all vision tokens in deep layers preserves high performance by relying on instruction tokens alone. In temporal video grounding, excessive dropping before the first attention layer (the 14th layer) harms performance, but the drop ratio can rise in later layers.

• Information flow differs by task type. For instruction-centric tasks (multi-choice video QA and temporal video grounding), blocking vision-to-instruction information (V2I) drastically degrades early-layer performance but has negligible impact in later layers. For vision-centric tasks such as detailed video captioning, blocking vision-to-response information (V2R) causes a sharp shallow-layer drop, showing that vision tokens directly drive response generation.

• TransV preserves grounding performance where dropping alone fails. Comparing table rows in the ablation, uniform token dropping at layer 7 drops Charades performance to 26.1, while using TransV at the same setting recovers it to 38.1, which the authors attribute to successful token information transfer.

• Shallow-layer placement is a tradeoff. Compressing at the 7th layer does not necessarily beat the 2nd layer: the 7th layer outperforms the 2nd by 0.6 points on multi-choice QA but performs 0.8 points worse on the detailed captioning benchmark.

• Attention-guided transfer helps in deep layers. Attention-guided TransV at layer 39 yields 56.6 on multi-choice QA versus 56.2 for uniform TransV, with minor differences on captioning and grounding. Transferring token information in deeper layers increases long-context capacity substantially: the model handles tens of thousands of frames with only a 0.1 drop on VideoMME.

• Aggressive compression costs accuracy. Raising the shallow-layer compression rate from 50% to 90% allows more frames but lowers VideoMME accuracy from 56.7 to 53.4.

• Memory and latency. The vanilla model runs out of memory at merely 128 frames. Adding ToMe in the projector extends the limit to approximately 5K frames, and TransV further extends it past 10K frames; at 4,096 frames, TransV reduces memory consumption by 54.8% compared to TimeViper (with ToMe only). On prefilling time, the vanilla model incurs 4.5s latency at 64 frames, which TimeViper reduces to 0.4s, and at 4,096 frames TransV reduces prefilling time by 15.7% relative to the ToMe baseline. TransV incurs no additional latency at low frame inputs (e.g., 64 frames).

• **

Authors’ abstract

We introduce TimeViper, a hybrid vision-language model designed to tackle challenges of long video understanding. Processing long videos demands both an efficient model architecture and an effective mechanism for handling extended temporal contexts. To this end, TimeViper adopts a hybrid Mamba-Transformer backbone that combines the efficiency of state-space models with the expressivity of attention mechanisms. Through this hybrid design, we reveal the vision-to-text information aggregation phenomenon, where information progressively flows from vision tokens to text tokens across increasing LLM depth, resulting in severe vision token redundancy. Motivated by this observation, we propose TransV, a token information transfer module that transfers and compresses vision tokens into instruction tokens while maintaining multimodal understanding capabilities. This design enables TimeViper to process hour-long videos exceeding 10,000 frames. Extensive experiments across multiple benchmarks demonstrate that TimeViper competes with state-of-the-art models while extending frame numbers. We further analyze attention behaviors of both Mamba and Transformer layers, offering new insights into hybrid model interpretability. This work represents an initial step towards developing, interpreting, and compressing hybrid Mamba-Transformer architectures.