Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding

AI summary

Overview

• Research area: Long-form video understanding with Large Multimodal Models (LMMs), specifically query-aware frame selection / video token reduction.
• Technical level: Intermediate. Readers should be comfortable with LMM inference pipelines, frame sampling, and embedding similarity, but the core argument is conceptual rather than mathematically heavy.
• Scope: The paper argues that frame-selection strategies should depend on query type, and introduces DIG, a training-free framework that uses uniform sampling for "global" queries and a multi-stage retrieval pipeline for "localized" queries.

What This Paper Is About

Feeding every frame of a long video into a Large Multimodal Model is infeasible because of limited context length and the sheer number of video tokens, so systems sample a subset of frames. Existing adaptive methods spend substantial computation searching for query-relevant frames, and the paper asks whether that expensive search is actually necessary for all queries. The authors show that it is not: splitting queries into global (whole-video) and localized (specific-moment) types reveals that simple uniform sampling already works well for global queries, while only localized queries genuinely need targeted selection.

Key Contributions

1. A query typology. The authors identify and validate a distinction between global queries (GQ), which require holistic video understanding, and localized queries (LQ), which target specific temporal segments, and show that frame-selection effectiveness depends heavily on this classification.
2. The DIG framework. A training-free frame-selection framework for LMMs that first uses an LLM to classify the query, then applies uniform sampling for global queries and a specialized pipeline for localized queries.
3. Content-Adaptive Frame Selection (CAFS). A method that uses pairwise similarity over DINOv2 features to pick semantically representative "r-frames" instead of static uniform or fixed-rate sampling.
4. Reward-guided video refinement. The LMM itself scores r-frames for query relevance, an iterative parameter-free selection keeps the above-average frames, and the corresponding segments are merged into a refined video before final uniform sampling. Experiments on three long-form benchmarks show consistent gains over uniform sampling, AKS, and Q-Frame, including at 256 input frames.

Main Findings

• More frames do not monotonically help. Across Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OneVision-7B on MLVU, VideoMME, and LongVideoBench, accuracy initially improves with more input frames but declines past a certain point.
• The degradation is query-type specific. When queries from MLVU and VideoMME were manually categorized, performance on global queries stayed relatively stable as frame count grew, while performance on localized queries dropped significantly, which the authors attribute to irrelevant frames injecting noise.
• Uniform sampling is sufficient for global queries. In the analysis, uniform sampling matched or occasionally exceeded the full DIG pipeline on global queries.
• The pipeline wins on localized queries. The proposed selection pipeline consistently outperformed uniform sampling on localized queries.
• Concrete accuracy gains at 32 frames. With 32 input frames, DIG improved Qwen2.5-VL-7B by 7.68% on MLVU and 4.51% on LongVideoBench compared with uniform sampling. The reported numbers at 32 frames for that model are MLVU 67.20, LVB 60.43, VideoMME Medium 61.62, and Long 53.24.
• Baselines can degrade at high frame counts. With Qwen2.5-VL-7B and 128 input frames, both AKS and Q-Frame underperformed uniform sampling by 1–2% on LongVideoBench, while DIG maintained gains.
• CAFS beats static sampling. CAFS outperformed uniform sampling and FPS sampling on the Global Coverage (GlC) and Localized Coverage (LoC) metrics, with the advantage most visible for videos longer than 10 minutes, and the gap widened as input frame counts increased.
• LMM rewards beat CLIPScore. Reward scores produced by Qwen2.5-VL-7B and Qwen2.5-VL-32B outperformed CLIPScore-based rewards in most reported cases, with the larger 32B model producing the strongest reward signals.
• Window length matters and is not monotonic. In tests of wlen ∈ {0, 2, 4, 8}, wlen = 0 gave the lowest performance (notably on LongVideoBench), while wlen = 8 degraded relative to wlen = 2 and 4; wlen = 2 was the best overall.
• Efficiency scaling. On LongVideoBench, uniform sampling saturated at a peak accuracy of 62.5%; DIG surpassed that peak once computation exceeded 720 TFLOPs (while operating at a budget of ≥ 680 TFLOPs) and kept improving.

Methodology in Plain English

DIG is a plug-in frame selection layer that sits in front of an LMM. It never trains anything.

1. Classify the query. An LLM first labels the query as global or localized.
2. Global queries: do nothing fancy. Just uniformly sample frames across the whole video and run the LMM.
3. Localized queries: search smartly.
   • A 2-fps version of the video is processed, and DINOv2 features are extracted per frame.
   • Consecutive-frame feature distances are computed as 1 minus cosine similarity. Sharp peaks in this distance sequence (where the value is greater than both neighbors, and prominence exceeds 0.1) indicate scene changes; these peaks become segmentation points.
   • One frame per segment — the midpoint — becomes an "r-frame," giving a compact summary of the video's semantic content.
   • The LMM itself scores each r-frame from 0 to 100, considering both direct usefulness for the query and whether nearby frames might hold supporting context.
   • A parameter-free iterative rule repeatedly subtracts the mean reward and keeps only frames still above zero, stopping when the selected set stops changing. This avoids a fixed Top-K hyperparameter.
   • For each surviving r-frame, the surrounding segment is retrieved, and neighboring r-frames within a window of length wlen are also merged in, producing a "refined video" that preserves fine-grained detail.
   • Final input frames are sampled uniformly from that refined video.

Why This Matters

Impact on research. The paper reframes a widely assumed premise in video-LMM work — that query-aware search is always the right answer. By showing that the benefit is conditional on query type, it suggests that many expensive selection pipelines may be spending compute where returns are diminishing, and it introduces two diagnostic metrics (GlC and LoC) for evaluating frame sets independently of downstream accuracy.

Real-world applications.

• Video surveillance and security review — answering questions about a specific event in hours of footage while still supporting broad "summarize the day" queries.
• Media and sports archives — locating a particular play, quote, or scene inside a long recording without processing every frame.
• Video editing and post-production — surfacing the relevant shots for a scripted question while still enabling whole-edit overview tasks.
• Educational and meeting recordings — question answering over lecture or meeting videos, where some questions are about one moment and others about the overall content.

Industry relevance. Context length and token cost are the binding constraints on deployed video LMMs. DIG is training-free, works with existing models (Qwen2.5-VL-7B and 32B in the experiments), and decouples the reward model from the inference backbone, meaning a stronger separate model can improve retrieval without replacing the serving model. That makes it an attractive inference-time upgrade rather than a retraining project. The reported scaling behavior — DIG continuing to improve past the point where uniform sampling saturates — is directly relevant to capacity planning.

Future Directions

• Replacing the LLM query classifier with something cheaper or learned. The current design routes each query through Qwen3-Next-80B-A3B for classification, and the authors note the efficiency gains from this module are analyzed in the appendix, implying cost trade-offs remain an active concern.
• Extending beyond pure visual input. The evaluation deliberately excludes subtitles, leaving open how audio, speech, and OCR signals would interact with the global/localized split.
• Adaptive frame budgets. DIG decides which frames to use but the experiments sweep fixed frame counts (8 to 256); tying the frame budget itself to the query and video length is not addressed.
• Generalizing the typology. The global/localized distinction is validated on MLVU and VideoMME with manual categorization, which raises the question of whether finer-grained or overlapping query categories exist and whether moderate wlen values generalize as well to other benchmarks and LMM backbones.

Target Audience

Researchers and engineers working on video understanding, video question answering, and multimodal LLM inference efficiency. It is also useful for practitioners deploying LMMs over long video at scale, since the method is training-free and can be added to an existing pipeline, and for students looking for a clear example of how a simple empirical observation (query types behave differently) can be turned into a full method and evaluation story. Background in vision-language models and frame sampling is assumed; the paper states its code is available at https://github.com/Jialuo-Li/DIG.

Authors’ abstract

The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.