Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs

AI summary

Overview

Research area: Computer Vision / video-language models — specifically long-form and streaming video question answering, efficient inference, and visual token compression.

Technical level: Advanced. The paper assumes familiarity with transformer attention, visual encoders, KV-caches, and video-LLM pipelines.

Scope: The paper presents a training-free, model-agnostic pipeline (rLiVS) that compresses short video clips into a small set of LLM-attended visual tokens, carries them forward recurrently, and answers questions from retrieved captions, evaluated on three offline and streaming video QA benchmarks.

What This Paper Is About

Video Large Language Models can answer questions about a video if they can see the whole video at once, but in streaming settings — where hour-long video arrives continuously and questions need timely answers — they cannot. The paper's goal is to let an existing short-clip video-LLM handle long streams without any retraining, by keeping only the visual tokens the model actually attended to, reusing those tokens across clips, and answering queries from stored text captions rather than by reprocessing old frames.

Key Contributions

1. rLiVS (Recurrent LLM-informed Visual Selection): a simple, training-free approach for long video understanding and question answering that selects visual tokens using the LLM's own attention to the caption it generated.
2. Architecture-agnostic design: the method requires no external modules and no retraining; it is demonstrated on LLaVA-OneVision (7B and 0.5B) and on Qwen2.5-VL-7B.
3. Recurrent memory over token selections: a FIFO queue of previously selected visual tokens is prepended to each new clip, giving past visual context to the current caption generation.
4. Caption-only retrieval and answering: questions are answered from stored clip captions retrieved with Maximal Marginal Relevance, which the paper reports as state-of-the-art performance with significantly lower memory requirements than prior streaming methods.

Main Findings

• Aggressive compression works: attention-based selection allows discarding up to ~95% of unimportant visual tokens with minimal performance loss. On RVS-style streaming, the pipeline retains 196 visual tokens from 3,136 available (16×196), i.e. 6.25% of total visual information per short clip.
• Attention beats naive selection on short video: on NextQA-valset, the full model scores 78.6 accuracy. At 6% of tokens, attention selection scores 77.0 versus uniform sampling 75.5, mean pooling 70.7, and K-Means 76.8. At 12%, attention scores 78.4 versus uniform 76.7, mean pooling 75.5, and K-Means 78.1. The paper states attention selection outperforms uniform sampling even at twice the compression rate, and that K-Means is significantly slower.
• Compression rate sensitivity: on NextQA valset, accuracy is 68.0 at 1% of tokens, 77.0 at 6%, 78.4 at 12%, 78.7 at 19%, 79.0 at 25%, and 78.6 at 100%.
• Offline long-video results (Table 1): with LLaVA-OV-7B, rLiVS reaches 61.0 Acc / 3.9 Sco on VS-Ego and 59.3 / 3.6 on VS-Movie, which the paper describes as outperforming the previous best by 2% and 3% respectively (Flash-VStream-7B: 59.0 / 3.9 and 56.1 / 3.4). On MovieChat it scores 78.0 / 4.0 (Goldfish 67.6 / 4.2, MovieChat 62.3 / 3.2). On CG-Bench it scores 33.1 (Chat-UniVi 25.9).
• Streaming results (Table 3): with LLaVA-OV-7B, rLiVS gets 65.3 / 4.0 on RVS-Ego and 57.7 / 3.6 on RVS-Movie at 1.9s latency and 25GB VRAM, versus ReKV on the same backbone at 63.7 / 4.0 and 54.4 / 3.6, 2.7s, and 36GB. The paper reports 11GB less peak VRAM and being nearly 1 second faster than ReKV, and describes this as outperforming ReKV by 2–3% on the same backbone.
• Small-model strength: with LLaVA-OV-0.5B, rLiVS scores 57.6 / 3.8 (RVS-Ego) and 51.3 / 3.4 (RVS-Movie) at 1.5s and 11GB, versus ReKV with the same 0.5B model at 54.7 / 3.7 and 44.6 / 3.4, 1.6s, 19GB, 4.0 GB/h KV-cache. The paper notes gains of 2.9% on RVS-Ego and 6.7% on RVS-Movie over ReKV at that size, and that the 0.5B variant outperforms the previous second-best approach using a 7B model (VideoScan at 60.9 and 54.1) — the paper states this for the 0.5B-versus-7B comparison generally.
• Model-agnostic gain: paired with Qwen2.5-VL-7B, rLiVS reaches 68.1 / 4.0 on RVS-Ego and 56.1 / 3.6 on RVS-Movie at 2.7s and 19GB.
• Recurrency matters: removing recurrency drops RVS-Ego from 65.3 / 4.0 to 62.5 / 3.9, RVS-Movie from 57.7 / 3.6 to 53.7 / 3.5, and MovieChat from 78.0 / 4.0 to 74.1 / 3.9 — a consistent 3–4% improvement from recurrency.
• Captions beat visual tokens for retrieval and answering (Table 5): on RVS-Ego, captions score 65.1 / 4.0, a combination 63.0 / 4.0, and selected visual tokens 58.2 / 3.9. On RVS-Movie, captions score 57.7 / 3.6, combination 54.3 / 3.5, and visual tokens 48.4 / 3.5.
• Attention beats uniform sampling in the streaming pipeline (Table 6): 65.1 / 4.0 versus 64.2 / 3.9 on RVS-Ego, and 57.7 / 3.6 versus 56.0 / 3.5 on RVS-Movie, a 1–2% margin with no additional overhead.
• Attention is instruction-sensitive: token selection varies with the task instruction, acting as a top-down signal; the paper reports this in Appendix H.

Methodology in Plain English

The pipeline splits a long video into short clips (16 frames each) and processes them in order.

Step 1 — Generate a caption for the current clip. The clip's frames are encoded into visual tokens by a vision encoder, projected into the language model's space, and the model writes a text description. To give the model context, a fixed number of visual tokens selected from earlier clips are prepended to the current clip's tokens.

Step 2 — Ask the model which tokens mattered. During caption generation, the model produces attention weights between its own generated caption words and the input visual tokens. The method averages these weights across caption tokens, across attention heads, and across a subset of layers (4 of 28 in their setup) to score every visual token globally. The top-scoring tokens are kept in their original temporal order.

Step 3 — Carry the winners forward. The selected tokens go into a FIFO queue of past selections (capacity 16 in the algorithm listing, with clip size 16). When the queue is full, the oldest selection is dropped. This queue is what gives later clips a compressed memory of earlier ones.

Step 4 — Answer questions from text. Each clip's caption is stored. When a question arrives, the method embeds the query and retrieves the top captions using Maximal Marginal Relevance — a score that balances similarity to the query against diversity among the already-selected captions, to reduce redundancy from recurrent captioning. Only the retrieved captions plus the question are passed to the LLM for the answer; no previous visual input is reprocessed and no external embedding encoder such as CLIP is needed.

Evaluation used a single A100 GPU, 10K context tokens for retrieval and generation, gpt-3.5-turbo-0613 for open-ended scoring, and frame rates of 0.5 FPS for RVS-Movie, RVS-Ego, CG-Bench, and offline VS-Stream, and 1 FPS for MovieChat.

Why This Matters

Impact on research: The paper argues that existing approaches either require training (with extrapolation problems on arbitrarily long videos and data-scarcity issues for long video annotations) or, like ReKV, store large KV-caches that become memory-intensive and redundant in streaming. It positions attention-based token selection plus caption-based retrieval as a strong, training-free baseline, and reports that retrieval and answering over captions effectively turns the problem into text-based long-context QA — a setting where LLM reasoning is well developed, unlike reasoning over very long visual contexts.

Real-world applications:

• Autonomous driving, where continuous visual input must be understood in real time.
• Surveillance, where streams run for hours and queries arrive online.
• Healthcare, where continuous visual monitoring requires timely responses.
• Entertainment and media, where long video content needs plot-level understanding and question answering.

Industry relevance: The affiliations (Toyota Motor Europe, University of Oxford, Athena RC) point to deployment-oriented settings where low memory and low latency matter. The reported reductions — 25GB versus 36GB VRAM and 1.9s versus 2.7s latency at the 7B scale, with no external memory offloading — are the kind of numbers that determine whether a streaming video assistant can run on available hardware.

Future Directions

1. Semantic rather than purely temporal memory selection: the FIFO buffer captures short-term context but may miss semantically salient information across long spans; the authors suggest exploring more semantic memory strategies that remain efficient.
2. Adaptive compression: dynamically adjusting token retention based on visual content complexity is proposed as promising, though it introduces computational overhead the authors currently avoid to preserve online efficiency.
3. Integrating into training pipelines: the method is training-free and inherits the limitations of pretrained backbones in visual understanding and temporal reasoning; the authors propose training-based integration and evaluation across a wider range of models and benchmarks.
4. Beyond question answering: the framework is proposed as a general-purpose backbone for long-form video summarization, retrieval, and object tracking, with preliminary summarization results in Appendix I.

Target Audience

Researchers and engineers working on video-LLMs, long-video and streaming video question answering, and efficient inference or token-compression methods. It is also relevant to practitioners deploying multimodal assistants in memory- and latency-constrained domains such as automotive, surveillance, and healthcare, and to those looking for a training-free baseline to compare against trained streaming approaches. Readers need a working understanding of transformer attention, visual encoders, and KV-cache mechanics.

Authors’ abstract

Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95% of unimportant visual tokens with minimal performance loss; 2) Recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.