Selecting or Solving? What Best-of-N Verification Delivers in Video Reasoning

AI summary

Overview

Research area: Computer vision and vision–language models (VLMs), specifically long-video question answering and inference-time compute (best-of-N selection with a verifier). Published at the VLM4RWD workshop (arXiv:2611.00507v1 [cs.CV], 31 Oct 2026).

Technical level: Intermediate. The paper assumes familiarity with vision–language models, chain-of-thought sampling, self-consistency (majority voting), and best-of-N reranking, but its central argument is conceptual rather than mathematical.

Scope (one sentence): The paper separates and measures two effects that a single best-of-N headline number conflates — the benefit a verifier adds by selecting among sampled answers versus the benefit it would have supplied anyway by solving the question itself — across five long-video benchmarks and a ladder of verifiers from weak open models to a frontier closed one.

What This Paper Is About

Answering a hard question about a long video usually turns on a few seconds of evidence buried somewhere in the footage, and VLMs remain well below human accuracy at grounding their answers in it. A standard way to spend extra test-time compute is best-of-N selection: sample several chain-of-thought answers from a fixed model, then have a second model re-watch the video and pick one. The problem is that a verifier which re-watches the video is unavoidably also a second solver, so the reported margin over majority voting cannot tell you whether the pipeline is selecting well or simply running a better model — and the right deployment decision differs sharply between the two.

Key Contributions

1. An audit protocol with two cheap controls. Strict candidates-only ("Selector") scoring, which credits a pick only if it is correct and drawn from the candidate set, plus the Selection Increment (SI) — selection accuracy minus the same verifier's accuracy with the candidates withheld. Neither requires retraining, and both are computed alongside the conventional margin over self-consistency (Δ_maj).

2. Two axes, two levers. SI falls monotonically as the verifier's standalone accuracy rises (the weakest verifier gains the most from seeing candidates), while beating the vote is governed by two conditions checked in order: whether the vote already sits near its ceiling (the saturation gate, roughly 87%), and only then whether the verifier is strong enough (the strength gate). The intuitive "headroom" rule — that selection helps most where the sampler leaves the most room above the vote — finds no support.

3. Two documented failure modes of scaled selection. Under-capable verifiers drift toward early answer positions rather than visual evidence (a threshold effect localized to a terse response mode), and augmenting candidate generation with an on-demand agentic evidence-gathering loop does not raise the oracle ceiling selection works against.

4. A reporting and deployment recommendation. Report both axes and the scoring policy behind each; skip verification where the vote is near the ceiling, otherwise pair the base with a clearly stronger cross-family verifier, and where that verifier still fails to beat the vote, deploy whichever cheaper alternative scores higher.

Main Findings

• The selectable gap is large but mostly unclaimed. On Video-Holmes (1,837 six-way reasoning MCQs over 270 videos), eight chains of thought from Qwen3.5-27B contain a correct answer 78.8% of the time (oracle@8), while self-consistency recovers only 58.9%. At N=16 the oracle reaches 82.5% while the vote stays flat at 59.0%.

• Only a strong verifier beats the vote on the anchor. GPT-5.6 Sol selects at 62.0% — +6.4 over pass@1 (55.6%) and +3.1 over self-consistency (58.9%, exact McNemar p = 7.0×10⁻⁴) — while leaving roughly 17 points of oracle headroom. All four weaker verifiers fall below self-consistency.

• The two axes dissociate, sometimes in opposite directions. On TempCompass (n = 1,574) the Muse verifier gains +7.94 points over the vote (79.2% → 87.2%) yet has SI = −1.08, meaning it answers better alone than when selecting. On VRBench (n = 983, majority 91.0%) it loses 3.26 points to the vote (p = 5.6×10⁻⁶) while SI is positive at +3.67. On MLVU-Test (n = 496) the margin is a non-significant +0.81 with SI = +5.68.

• Inside a single candidate pool, the axes come apart completely. On the Video-Holmes Qwen-base pool, the weak Phi-3.5 verifier has SI = +19.3 from a 32.5% standalone accuracy, yet its selected accuracy (51.8%) lands 7.1 points below the majority vote (58.9%, p < 10⁻¹³). The strong Muse verifier adds almost nothing over its own solving (SI = +2.5) and significantly beats the vote (+2.8, p = 2.3×10⁻³).

• SI tracks standalone accuracy inversely, monotonically. The five-verifier ladder on Video-Holmes runs +28.7 (Qwen-27B self-verifier, standalone 27.7%), +19.3 (Phi-3.5, 32.5%), +17.6 (InternVL3-8B, 39.7%), +10.9 (InternVL3-38B, 47.7%), +7.9 (GPT-5.6 Sol, 54.1%). An MLVU-Test ladder reproduces the ordering (+26.3, +21.6, +20.8, +10.8 as standalone rises from 38.6% to 46.7% to 51.1% to 62.0%). A large increment therefore marks a weak solver that can still select, not a good verifier.

• Saturation is checked before strength. Above roughly 87% majority accuracy no verifier tested beats the vote; on VRBench both deployed verifiers significantly lose. Below the ceiling, strength decides: on the Qwen-base Video-Holmes pool (58.9%) only the two strongest verifiers win (+2.8 and +3.1) and every weaker one fails. Holding the task fixed, the Muse verifier's +7.94 win on TempCompass collapses to −0.4 as majority accuracy rises from 79.2% to 87.5%.

• One null the authors do not explain. MLVU-Test sits at roughly 73% majority accuracy with a capable verifier, and still neither deployed verifier beats the vote (+0.81 and −1.99, both non-significant). The ceiling itself rests on two near-saturated pools, so the paper reports an observed boundary rather than a calibrated threshold.

• Weak verifiers rubber-stamp or drift by position. On the Qwen-base TempCompass pool (79.2%) the weak self-verifier only draws level (+0.9, p = 0.082) and agrees with the plurality on 95.1% of rows. On EgoSchema (five-way, n = 500, near-uniform gold whose mode is E at 23.4%), Phi-3.5-Vision shows A-lift of +4.4 at 16 frames and +8.0 at 48 frames, picking the modal answer E only 13.4% of the time; every capable verifier stays within 1.4 points of gold.

• Position bias is a threshold, not a gradient. Rank correlation between accuracy and A-lift is ρ = −0.954 across ten configurations (exact permutation p = 4.5×10⁻⁵ over all 10! label assignments); dropping the weak Phi-3.5-Vision verifier moves the pooled within-benchmark test from p = 0.013 to p = 0.34. Of 22 verifier–benchmark cells, nine are individually significant by exact McNemar and five survive Holm–Bonferroni; all four positive survivors belong to Phi-3.5-Vision.

• The bias lives in a degenerate response mode. The weak verifier emits either a "Answer: X" marker mode or a bare-letter terse mode, separated perfectly by output length. Marker-mode responses show no significant A-lift in 10 of 10 arms; terse-mode responses favor the first option in 8 of 8 arms where the mode occurs (+6.7 to +32.4 points, largest p = 2.5×10⁻³, all surviving Holm–Bonferroni over eighteen mode tests). On EgoSchema at 16 frames the two modes are equally accurate within 0.11 points, so the default is directional rather than a by-product of solving worse.

• Gathering more evidence does not raise the ceiling. Adding an agentic loop that may zoom, crop, retrieve frames, or ground objects leaves the best-of-N oracle on EgoSchema unchanged (Δ_oracle = 0.000 at n = 500), and giving the same loop to a strong cross-model verifier leaves its accuracy within noise of its zero-shot self.

• A genuine, surviving gain. A cross-model verifier does beat self-consistency by +7.94 points on TempCompass. The strongest Video-Holmes margin does not survive strict scoring after multiplicity correction, and the authors explicitly decline to claim it.

Methodology in Plain English

The setup is deliberately simple. A fixed base model (Qwen3.5-27B) watches uniformly sampled frames (64, or 32 on EgoSchema) at temperature 0.8 and produces N = 8 independent chain-of-thought answers to each multiple-choice video question, each ending in a candidate letter. Three rules then score that same pool of candidates: majority voting, a second VLM (the verifier) that re-watches its own K frames and makes one holistic N-way pick, and — crucially — that same verifier answering the question without seeing the candidates.

The last of these is the key move. Because the verifier can be run both with and without the candidate list, the authors can subtract its own solving ability from its selection accuracy, isolating exactly what the candidates contributed. Two further controls prevent the comparison from flattering the verifier twice over: a strict scoring policy that only counts a pick if it is both correct and actually drawn from the candidate set (since a verifier can ignore the list, emit its own letter, and still be credited under conventional scoring), and matched-sample statistical testing (exact two-sided paired McNemar on discordant pairs, with Holm–Bonferroni correction where families of comparisons are named).

The verifier ladder spans model family and capability: Phi-3.5-V (weak), InternVL3-8B and -38B (mid), Qwen3.5-27B and Qwen3.5-397B (same-family, the latter 15× the base's size), Muse Spark 1.1, and GPT-5.6 Sol, over frame budgets K ∈ {16, 24, 32, 48, 64, 96}. The five benchmarks were chosen so that self-consistency accuracy spans a wide range, from about 59% (Video-Holmes) to about 91% (VRBench), testing selection in easy and hard regimes alike. Position bias is measured with two statistics — A-lift and primacy — that agree closely across 22 verifier–benchmark cells (Spearman +0.942). Two independently implemented scorers recomputed every headline number, agreeing to within 0.1 points.

Why This Matters

Impact on research. Best-of-N pipelines are routinely reported as a single margin over majority voting, and this paper shows that number is compatible with two opposite mechanisms and two opposite deployment decisions. The Selection Increment is a cheap, training-free audit that any lab can run before committing inference compute, and it belongs to a broader audit literature asking whether reported gains reflect added capability or artifacts of format and position. The finding that a large increment can belong to a verifier that loses to a majority vote is a direct caution for how these results get ranked and reported.

Real-world applications:

• Long-video and egocentric QA assistants, where a user asks a complex question about hours of footage and the system must justify an answer from a few seconds of evidence.
• Media archive and broadcast search, where multiple-choice or candidate-constrained answers must be grounded in specific moments rather than plausible summaries.
• Video review in surveillance, industrial inspection, or clinical recording review, where the cost of a wrong answer is high and a confidently position-biased picker is worse than a vote.
• Any deployed system routing between a cheap base model and an expensive second model, where the decision of whether to call the verifier at all has a direct cost consequence.

Industry relevance. The three-step allocation policy is directly operational: compute the majority accuracy on a labeled development slice before calling any verifier, skip verification where the vote is near the observed ceiling, and where a strong cross-family verifier still does not beat the vote, deploy whichever of the two cheaper alternatives measured higher. The paper also notes the practical caution that same-family verifiers may share the base's errors, motivating cross-family pairing.

Future Directions

• Explain the MLVU-Test null. At roughly 73% majority accuracy with a capable verifier, neither deployed verifier beats the vote, and the authors state they cannot presently explain it.
• Calibrate the saturation threshold. The roughly 87% boundary rests on two near-saturated pools, so it is an observed boundary rather than a calibrated one; more pools spanning a wider headroom range are needed, since the current ≈6–14-point range supports only "no evidence for the headroom rule."
• Separate ordinal position from letter rank. Every benchmark used labels options alphabetically, so the two are collinear and the position-bias statistics establish the drift without disentangling the causes.
• Extend beyond final answer letters and uniform frames. The protocol selects over answer letters rather than intermediate reasoning steps, and uses uniformly sampled rather than query-conditioned frames; a benchmark-matched token accounting for verification is also left to future work.

Target Audience

Researchers and engineers working on vision–language models, test-time compute, and inference-time scaling for video or multimodal reasoning — particularly those building or evaluating best-of-N verifier pipelines and deciding whether a second model is worth its cost. It is also relevant to evaluation and benchmarking researchers interested in audits of reported gains, and to practitioners who need a cheap pre-deployment check before allocating inference budget. Readers without background in sampling-based decoding will find the core argument accessible but the experimental framing dense.

Authors’ abstract

Answering a complex question about a long video is a grounding problem: the answer turns on a few seconds of evidence, and vision-language models remain well below human accuracy at finding them. The standard way to spend extra test-time compute on this kind of video reasoning is best-of-N selection: sample several chain-of-thought answers from a fixed model, then let a second model re-watch the video and pick one. Such pipelines are reported by a single headline number, their margin over a majority vote. That number conflates two effects with opposite deployment consequences: the benefit the second model adds by selecting among candidates, and the benefit it would have supplied by solving the question itself. We separate them, measuring the Selection Increment (SI) -- selection accuracy minus the same verifier's accuracy with the candidates withheld -- alongside the margin over self-consistency, across five long-video benchmarks. The Selection Increment is largest for the weakest verifiers: a verifier that solves poorly on its own leaves the candidates the most room to help. Beating the majority vote follows the opposite profile: where the vote already sits near its ceiling no verifier we test improves on it; below that ceiling only the strongest do. On Video-Holmes, the two come apart on a single pool of candidates -- a weak verifier gains +19.3 points from the candidates yet lands 7.1 points below the vote, while the frontier verifier gains the least and beats the vote. The weakest verifier also shows a failure mode: on EgoSchema it picks by where an option sits in the list rather than by what the video shows. Read as a merit score, SI puts a verifier that fails to beat a majority vote at the top of its ranking; read as an audit, it says which of the two effects a headline number delivers. Code is available at https://github.com/facebookresearch/best-n-selection-video-reasoning.