FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

AI summary

Overview

Research area: Computer Vision / multimodal large language models — specifically keyframe selection for long-video question answering with VideoLLMs.

Technical level: Advanced. The paper assumes familiarity with transformer KV caching, contrastive learning, and VideoLLM tokenization, though the core ideas are explained in reasonably self-contained terms.

Scope: FrameScout is a lightweight, learned frame selector that picks the 60 most query-relevant frames from videos of tens of minutes to hours, using a streaming architecture with constant memory and a frame–query contrastive objective.

What This Paper Is About

VideoLLMs cannot ingest long videos because visual tokens grow linearly with length: the paper notes that a 20-minute video sampled at 1 frame per second, with 256 visual tokens per frame under the Qwen2.5-VL encoder at 448×448 resolution, yields roughly 307K tokens — far beyond Qwen2.5-VL-7B's 32K context. The standard fix is to select a small set of query-relevant keyframes, but existing selectors either score frames independently without temporal context (FOCUS) or process chunks in isolation with a ranking objective that is hard to optimize (FlexSelect). FrameScout's goal is a selector whose frame embeddings capture context from both before and after each frame, and whose training objective directly optimizes the relevant-vs-irrelevant decision made at inference.

Key Contributions

1. A streaming selector architecture. A lightweight VideoLLM (vision encoder, language model, two-layer MLP projector) processes the video in non-overlapping chunks while carrying a sliding-window KV cache — a fixed system prompt of 256 tokens kept as attention sinks plus the KV entries of the W most recent chunks, re-indexed with RoPE. Total cache size stays constant at |s| + WBL tokens regardless of video length.
2. Successor aggregation for future context. A causal KV cache only sees the past, so the method adds a discounted backward aggregation, M_i = φ(Σ β^k h_{i+k}) with β = 0.3, computed by a single backward pass via the recurrence z_i = h_i + β·z_{i+1}. This gives bi-directional contextualization at O(Nd) extra computation and O(d) extra memory.
3. A frame–query contrastive objective. Frames and queries are embedded into a shared space, and a contrastive loss (temperature τ = 0.05) pushes top-k pseudo-labelled positive frames toward the query and all remaining frames away, replacing FlexSelect's Spearman rank-correlation loss.
4. Large-scale evaluation. Three long-video benchmarks (VideoMME, MLVU, LongVB), six base VideoLLMs spanning closed- and open-source models, with reported state-of-the-art results and multi-query speedups.

Main Findings

• Consistent gains over uniform sampling: FrameScout improves every model–benchmark combination in Table 1. With Qwen3-VL-8B it gains +6.5 on VideoMME-Long, +4.7 on MLVU, and +4.9 on LongVB. With Qwen2.5-VL-7B it gains +5.9 on VideoMME-Long and +5.4 on LongVB. With GPT-5.4 it lifts VideoMME Overall by +5.6 and MLVU by +3.5.
• Beats the state-of-the-art selectors: On Qwen3-VL-8B, FrameScout exceeds FlexSelect by +1.6 on VideoMME Overall, +1.0 on MLVU, and +3.7 on LongVB; the margin over FOCUS is described as even larger.
• Baselines are unstable, FrameScout is not: FOCUS degrades performance in several settings (−6.7 on VideoMME-Long with GPT-5.4, −6.6 with LLaVA-Video-7B), and FlexSelect harms some configurations (−3.8 on MLVU with GPT-5.4, −1.7 on VideoMME-Long with LLaVA-Video-7B). FrameScout delivers positive gains in all reported settings.
• Multi-query efficiency: Because frame embeddings are query-agnostic and computed once per video, total latency stays nearly flat as queries grow from 1 to 8. At eight queries FrameScout achieves a 21.2× speedup over FOCUS and 6.3× over FlexSelect. Per-query selection costs roughly 0.03 s.
• Constant memory: Peak memory of frame selection stays around 8 GB from 64 to 4096 frames, while FlexSelect exceeds 80 GB at 4096 frames — about 10× higher. Storing and reloading embeddings for an 8192-frame video takes 35.75 ms, only 0.014% of its 252 s encoding time.
• The streaming design matters: Replacing streaming with independent, non-overlapping windows drops VideoMME from 65.6 to 65.0 and LongVB from 61.3 to 63.3 on Qwen2.5-VL-7B; for LLaVA-Video-7B it drops 65.7→63.7 and 63.4→60.6.
• Representations stay faithful: On 100 videos of 1024 frames each, streaming outputs have per-video mean frame-level cosine similarity tightly concentrated near 1.0 against full-sequence inference, while separate windows show substantially lower similarity and larger variance.
• Training on isolated chunks is better: Under the same wall-clock budget, training with independent chunks (plus the sink cache) gives training loss 0.2057 and VideoMME 65.6 / LongVB 63.3, versus 0.3731, 64.2, and 62.1 for training with the streaming KV context.
• Backbone-agnostic: Swapping InternVL2.5-1B for the smaller LLaVA-OneVision-0.5B still beats uniform sampling (55.8 vs 51.4 on VideoMME-Long, 64.6 vs 62.6 Overall), with the larger selector performing better.
• Well-separated embedding space: In a t-SNE plot on a NExT-GQA sample, the query embedding sits closer to ground-truth keyframes (average cosine similarity 0.33) than to background frames (0.12), and the top-3 retrieved frames fall inside the ground-truth span.
• Note on missing numbers: The paper states that comparisons with BOLT, F2C, M-LLM, HICom, and VTM appear in Appendix B.2; those figures are not in the content provided here.

Methodology in Plain English

The selector is a small VideoLLM, deliberately much cheaper than the model that ultimately answers the question — otherwise selecting frames would cost more than just answering directly.

Encoding the video. The video is chopped into chunks of 12 frames. Each chunk is fed to the language backbone together with a compressed memory of what came before: a fixed 256-token system prompt whose cache is never discarded (it keeps attention from becoming unstable, following the StreamingLLM attention-sink idea) and the cached keys/values of the most recent chunks. As new chunks arrive, the oldest chunk's cache is evicted and positions are re-indexed. Because the hidden state of every earlier chunk already absorbed its own history, this chain effectively lets each frame "see" everything before it, without ever holding the whole video in memory.

Looking ahead. A causal cache cannot see the future, but relevance often depends on what happens next. The authors borrow the successor-representation idea from reinforcement learning: walking backward from the last frame, each frame's summary becomes its own hidden state plus a decayed copy of the next frame's summary (β = 0.3, so nearby frames weigh more). One backward pass, negligible cost, and every frame embedding now encodes both past and future.

Training. Frame-level relevance labels do not exist in video QA datasets, so the authors generate them: 100 frames are fed to a teacher (InternVL2.5-8B), and the attention weights from query tokens to visual tokens, averaged over heads and pooled per frame, become relevance scores. The top-k frames form the positive set, everything else the negative set. The selector — sharing the same backbone and projector for queries and frames — is trained with a contrastive loss that pulls positives toward the query and pushes negatives away. Training uses independent chunks with only the sink cache, which allows chunk-level parallelism and bounds activation memory; the streaming cache is switched on at inference, and the ablation shows this is both faster to converge and more accurate.

Inference. Encoding happens once per video; each query then needs one text forward pass and N dot products, followed by a top-K selection (K = 60 for every method and setting, so comparisons isolate selection quality). The selected frames go to whatever base VideoLLM the user has, which can be swapped freely.

Why This Matters

The paper targets a hard structural limit rather than a benchmark quirk: context windows grow, but hour-scale video at 1 fps saturates even a 1M-token window, and densely prefilling such contexts is expensive in compute, memory, and latency. Making selection cheap, query-agnostic, and reusable changes the economics of long-video QA — particularly for multi-turn sessions about the same video, where existing selectors must re-encode everything for each new question.

Real-world applications:

• Reviewing long surveillance or security footage with natural-language questions, where constant memory per video length is a deployment requirement.
• Querying recorded lectures, meetings, or conference talks, where a user asks several follow-up questions about one recording.
• Searching film, sports, or archival footage for specific moments described in text.
• Question answering over long procedural or medical video, where the relevant evidence spans several minutes and requires temporal context to interpret.

Industry relevance: The memory profile is the headline for practitioners — around 8 GB regardless of whether the video is 64 or 4096 frames, versus FlexSelect crossing 80 GB at 4096 frames — meaning hour-scale selection can run on a single GPU. Decoupling the selector from the answering model also means a company can upgrade its VideoLLM without retraining the selector, and the one-time encoding plus ~0.03 s per query fits interactive products.

Future Directions

• Reliance on teacher attention: Pseudo-labels come from InternVL2.5-8B's attention weights, so the selector may inherit the teacher's blind spots. Whether stronger or ensembled teachers improve selection is untested here.
• Window and discount sizing: Chunk size B = 12 and β = 0.3 were chosen for an accuracy–efficiency trade-off; the paper defers sensitivity analysis to Appendix B.1, and it is unclear how these should scale for videos far longer than those benchmarked.
• Beyond keyframe selection: The selector returns K = 60 frames with no notion of coverage or redundancy among them, and no adaptive per-query budget — an obvious extension given that "how many frames" is fixed at 60 across every setting.
• Training/inference mismatch: The streaming cache is only active at inference; closing that gap without losing chunk-level parallelism, or showing that the gap stays harmless at longer horizons, remains open.

Target Audience

Researchers and engineers working on long-context multimodal models, video retrieval, or efficient VideoLLM inference. It is most useful to readers who already understand transformer KV caching and contrastive objectives and want a concrete, well-ablated recipe for frame selection. Practitioners deploying video QA on fixed GPU budgets will find the memory and multi-query latency results directly actionable; newcomers to the area should read the introduction's framing of the token explosion problem first, then consult the cited baselines (FOCUS, FlexSelect) for context.

Authors’ abstract

Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current VideoLLMs can process. To address this issue, keyframe selectors are proposed to select the most query-relevant frames by computing the similarity between frame and query embeddings. Existing selectors either encode frames independently without temporal context or process chunks in isolation with ranking objectives that are difficult to optimize. To alleviate those limitations, we propose FrameScout with two core designs: (i) a streaming selector architecture that leverages a sliding-window KV cache and successor aggregation to produce temporally-contextualized frame embeddings; (ii) a frame-query contrastive objective that aligns frame and query embeddings in a shared space, directly training the selector to distinguish relevant frames from irrelevant ones. Extensive experiments on three long-video benchmarks and six base VideoLLMs demonstrate that FrameScout consistently achieves state-of-the-art performance. Code is available at https://github.com/Hubert-1102/FrameScout.