Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically long-video question answering and the visual-token budget problem.

Technical level: Advanced. The paper is written for readers comfortable with paired statistical testing (McNemar tests, two-one-sided tests), sparse-approximation algorithms, contrastive image-text embedding geometry, and video-language benchmark design. The core concepts are explainable in plain language, but the evidence standard is quantitative and dense.

Scope: A single-author controlled study, arXiv:2609.03820v1 [cs.CV] (03 Sep 2026), that isolates three decisions in long-video MLLM pipelines — which frames to select, how far to compress each frame, and how to reinvest the saved budget — while holding the scorer, prompt boundary, frame budget, and answering model fixed.

What This Paper Is About

Long-video language models cannot look at every frame: an hour of video sampled at one frame per second is 3,600 images, and the system keeps only a small fixed slice of that pool (eight frames in some systems, thirty-two or sixty-four in others). Published frame-selection methods are hard to compare because a new selector usually arrives bundled with a new scorer, a different prompt, a different resolution policy, and a different answering model, so a reported gain could come from any of those.

This paper holds all of those components fixed and varies one decision at a time across six training-free selection rules, three long-video benchmarks, and two answering models. The goal is to determine which selection rule supplies the most useful evidence under matched conditions, whether compressing frames costs accuracy, and whether spending the freed budget on more frames beats spending it on sharper ones.

Key Contributions

1. A matched comparison of six training-free selectors under one scorer (LongCLIP), one prompt boundary (question stem only), one frame budget (k = 8), and one answerer harness. Orthogonal Matching Pursuit (OMP) stays within one point of LDDR's stage-1 selector across all three benchmarks.

2. Evidence that selection substitutes for frame count. OMP significantly outperforms uniform sampling while using half as many frames, in two disjoint LongVideoBench duration bins (600 s and 3600 s) that share a benchmark, scorer, answerer, and implementation.

3. A decomposition of fixed-budget allocation. Roughly halving the per-frame spatial budget preserves accuracy, bounded by a two-one-sided-test (TOST) interval rather than a bare null result, at a margin chosen post hoc; reinvesting the savings in additional keyframes then improves long-video QA.

4. A demonstration that aggregate accuracy can conceal implementation error. A padding branch in the authors' own AKS port made that baseline select global top-k at every k < 32. Correcting it moved roughly 99.5% of selected frames yet changed LVBench accuracy by 0.07 points.

5. Explicit boundaries on the claims, from paired tests, two answerer families, three benchmarks, a residual-geometry diagnostic, and a purposive failure audit. The paper states that these boundaries are uneven: the scorer swap covers one bin, the equivalence result one pooled setting, and the audit is exploratory.

Main Findings

• Selection is the largest lever. OMP improves on uniform sampling by 5.69 points on LongVideoBench, 5.85 on Video-MME, and 11.81 on LVBench at k = 8, with paired significance on Video-MME (p = 5.1 × 10⁻¹⁰), LVBench (p = 9.5 × 10⁻¹⁷), and the LongVideoBench 3600 s bin (p = 5.9 × 10⁻⁴). The matched-selector table gives OMP .6223 / .6222 / .4635 (LongVideoBench / Video-MME / LVBench), with LDDR-select at .6320 (+0.97) / .6193 (−0.29) / .4693 (+0.58).

• Half the frames can beat double the frames. On 3600 s LongVideoBench videos, OMP with eight frames reaches .5461 while uniform sampling with sixteen reaches .4770, a 6.9-point gain on half the input (p = .0011). On 600 s videos, OMP with sixteen frames reaches .6578 against .6044 for uniform sampling with thirty-two, a 5.3-point gain at half the frame count (p = .018).

• The effect switches on with candidate-pool size, not duration. On 15 s clips, uniform, top-k, and OMP return exactly the same accuracy (.7249). Gains appear at 600 s (7.8 points, p = .0022) and 3600 s (7.5 points, p = 5.9 × 10⁻⁴). The gain does not keep growing: 7.5 at 3600 s is slightly below 7.8 at 600 s, so this is a threshold rather than a duration gradient. The authors note the bins differ in content and question composition, so this is an observational contrast.

• Compression is close to free. Fixed timestamps with roughly half the per-frame budget shift accuracy by at most 0.44 points across Qwen benchmark aggregates (LongVideoBench +0.30, Video-MME −0.44, LVBench +0.39), with GPT-5-mini on LVBench at +0.26 (p = .84) and a uniform-sampling control at +0.31 (p = .784). On pooled LongVideoBench long bins (n = 976), the compressed OMP arm is 0.72 points higher with a 90% CI of [−0.60, +2.03], which fits a ±3-point margin (TOST p = .0022) but narrowly fails at ±2 (p = .054). A ±3 margin is reported as "the honest statement."

• The shape of the compression schedule matters. The residual-proportional schedule does not beat a flat split at the same mean budget. A reconstruction of LDDR's stage-2 Group-DPP importance improves on the flat split by 1.84 points on pooled long bins (p = .0198, 95% CI [+0.37, +3.32]), though the 3600 s component is not individually significant, so this is treated as indicative.

• Reinvestment converts the free budget into 2–3 points. Sixteen compressed frames beat eight full-resolution frames: LongVideoBench +2.24, pooled 600/3600 s bins +2.36 (p = .0346), Video-MME +1.56, LVBench +3.04 (p = .0009), GPT-5-mini on LVBench +2.39 (p = .039). The sixteen-frame arm consumes 0.996 and 0.984 times the tokens of the eight-frame full-resolution arm in the 600 s and 3600 s bins, so the winning arm is also the cheaper one. No separate token audit was run on Video-MME or LVBench.

• Reinvestment is not shown to require OMP. In a single-environment four-cell design, uniform sampling gains +1.13 points (p = .3673) and OMP gains +2.56. The interaction is 1.43 points and not significant (p = .2614), with OMP gaining on 110 items where uniform does not and uniform gaining on 93 where OMP does not. Split by duration: +0.00 at 600 s (p = 1.000) and +2.48 at 3600 s (p = .1706). The authors state the selector-specificity remains open at roughly the one-point scale.

• The selector ranking survives a scorer swap. Replacing LongCLIP with SigLIP-so400m changes 67–84% of the frames the answerer sees (the two scorers agree on only 1.32 of eight frames for OMP and 2.62 for top-k on the 600 s bin, seven to fourteen times chance overlap). Yet the ordering — uniform beaten by top-k, top-k beaten by OMP — holds under both. OMP's advantage over uniform is significant under each (p = .0024 LongCLIP, p = .00026 SigLIP). The test is not powerful: with roughly 134 discordant pairs, the 90% interval on the OMP scorer effect spans about [−3.6, +5.6] points, which bounds large scorer effects rather than establishing their absence. The authors also caution against reading a significant SigLIP result beside a non-significant LongCLIP one as an interaction: the difference of the two gains is 0.97 points (p = .68).

• The prompt boundary is an uncontrolled axis worth 2–4 points. Scoring against the fused question-and-options string instead of the stem alone changes 41.9% of top-k's frames on LongVideoBench-600 s and 53.3% of OMP's. OMP reaches .6699 against .6311 for the stem (+3.88 points, p = .033, 33 rescued against 17 broken); top-k reaches .6408 against .6068 (+3.40, p = .087). A pre-registered replication on the 3600 s bin shows OMP at .5691 against .5461 (+2.30 points, n = 564, 46 rescued against 33 broken, p = .18), with the fused query moving roughly 69% of selected frames. A control replacing each item's options with those of a different video reaches .6529, recovering 56% of the gain, which rules out that the effect belongs to the answer set. The paper deliberately scores with the stem alone, following the official AKS implementation, so every selector is evaluated in a conservative configuration.

• Later OMP picks do not behave as sparse reconstruction predicts. On both long bins, the residual norm is 0.972 of its starting value after one pick and 0.967 after eight, and does not fall further by sixteen. Residual correlation with the selected frame drops from .233 at pick one to .004 at pick eight and to roughly zero by pick sixteen, yet later frames keep a cosine near .20 with the original query. The authors offer this as a hypothesis consistent with the discrepancy, not a demonstrated mechanism, and note nothing in the study intervenes on residual geometry.

• Failures cluster on off-topic novelty. An exploratory inspection of 93 OMP failures at k = 8 across the two long bins found this pattern in 20 of 41 inspected 600 s cases and 35 of 52 inspected 3600 s cases: several picks went to visually distinctive but question-irrelevant material such as unrelated chapters, title cards, and dark transitions. The mirror-image failure also appeared, where dense top-k sampling clustered on one relevant moment and missed another required scene.

• The same frames do not help every model equally. With identical OMP and uniform timestamps, InternVL3-2B and InternVL3-8B both replicate the selection gain on LongVideoBench (+4.71 and +6.51) and LVBench (+9.04 and +10.14). Video-MME does not cooperate: the 2B model benefits at every duration (+6.00 short, +3.33 medium, +4.22 long) while the 8B model benefits only on short videos (+5.44) and is flat on medium (+0.33, p = .891) and long (+0.11, p = 1.000), despite receiving demonstrably different frames. InternVL3-8B holds accuracy through a threefold tile reduction (−0.10 points, p = 1.000) and gains 3.38 points when the same total tile budget is spread from 8 uniformly sampled frames to 24 (p = .0099), an effect concentrated in the 600 s bin.

• Small implementation details can matter more than large frame changes. The authors' AKS port had a padding bug that made the baseline select global top-k at every k < 32; fixing it moved roughly 99.5% of selected frames but changed LVBench accuracy by only 0.07 points. A separate observation: two harnesses running the same published rules at the same budget differed by 0.07–3.74 points, which is the paper's stated reason for keeping comparisons inside one controlled harness.

Methodology in Plain English

The authors treat every comparison as a paired experiment: one benchmark question answered twice, once under each of two input policies, with everything else held constant. This lets them count which questions each policy wins rather than compare two accuracy numbers that might differ for unrelated reasons.

Three interventions are run in sequence:

1. Select — change which timestamps are chosen, holding the frame count and resolution fixed.
2. Compress — hold those timestamps fixed and shrink the spatial budget spent on each frame.
3. Reinvest — spend the recovered budget on more timestamps rather than sharper ones.

A fourth comparison sits outside the sequence: swapping the scorer (LongCLIP for SigLIP-so400m) while holding the selection rule, budget, and answerer fixed, to check that the ordering is a property of the rules rather than of one encoder.

Videos are decoded at 1 fps. Each candidate frame and the question stem are encoded once with LongCLIP, and the embeddings and stem similarities are cached. Every selector reads the same cache, so no rule benefits from a better encoder than another. The six rules compared are uniform sampling, cosine top-k, AKS, FOCUS⋆, OMP, and LDDR-select. The authors note that AKS and FOCUS are reproductions of a subset rule under a substituted scorer (both natively use BLIP-ITM), LDDR-select is the only row evaluated with its authors' encoder, and FOCUS⋆ replays the published clip-bandit schedule against dense LongCLIP scores rather than the original budgeted online ITM process.

The central algorithm, Orthogonal Matching Pursuit, is an unmodified greedy sparse-approximation method from 1993: at each step it takes the candidate frame most correlated with what the query still needs, then projects that direction away before choosing again. The authors frame this as a methodological choice rather than a concession, because an off-the-shelf rule has no hyperparameters to fit to these benchmarks.

For compression, frames are resized before reaching the model's processor, with Qwen3-VL's internal resize disabled so the authors' resize is the only one applied. The main compressed arm uses a residual-proportional schedule targeting a mean spatial fraction of about 0.53, labeled D@53. Two controls test the schedule's shape: a flat split at the same mean fraction, and a reconstruction of LDDR's stage-2 Group-DPP importance applied to identical OMP-8 timestamps.

For reinvestment, the arm selects k = 16 timestamps with an approximately 50% per-frame cap. Because the comparison only means something if the two arms really cost the same, the authors audit tokens empirically with a script that reproduces Qwen's smart-resize rules from each video's own dimensions.

Statistics use exact two-sided McNemar tests on paired correctness outcomes. Where the authors claim two arms are interchangeable, they additionally report two one-sided tests with 90% confidence intervals, which bound the effect rather than merely failing to find it. The ±2, ±3, and ±4 percentage-point margins were chosen after seeing the runs rather than preregistered, so both the narrowest supported margin and the adjacent unsupported one are reported. Tests are contrast-specific and uncorrected for multiplicity.

Evaluation runs on LongVideoBench (complete official validation set, n = 1337, stratified into 15, 60, 600, and 3600 s bins), Video-MME (n = 2700), and LVBench (n = 1549). The primary answerer is Qwen3-V

Authors’ abstract

Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.