Supervising the Visible State: Separating Representation from View Selection in Streaming Video

AI summary

Overview

Research area: Streaming video understanding for vision-language models (video LLMs), specifically temporal post-training and inference-time evidence selection.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, multimodal backbones, confidence margins, and benchmark-based evaluation.

Scope: The paper proposes and evaluates two separable interventions—Visible-State Supervision (ViSS) for training and Test-Time Causal View Search (TVS) for inference—on a single Qwen3-VL-8B backbone across StreamingBench and OVO-Bench.

What This Paper Is About

Streaming video models answer questions from only the observed prefix of a video, under bounded context and computation, so their accuracy depends on what history the model has retained. The authors argue that ordinary answer-level supervision under-determines this history: one repetition and three repetitions of the same action get the same action label, leaving counting, order, and object-relation distinctions unconstrained—a problem they call state under-specification. The paper separates two levers that are usually conflated: what supervision asks the model to represent (ViSS) and what evidence inference exposes at query time (TVS), measuring each with controlled interventions on one backbone and the same recent-four-frame starting view.

Key Contributions

1. Identifying state under-specification. The authors name and formalize the gap in answer-level supervision, where histories differing in event order, accumulation, or relations map to the same target and therefore leave those distinctions unconstrained.

2. Visible-State Supervision (ViSS). A 7,677-example source-grounded training mixture of probes covering event identity, order, accumulation, relation change, and answer sufficiency, trained for one epoch. Under the same recent-4 interface it beats a strict-matched generic temporal SFT control by +1.73 points on OVO-Bench and +8.06 points on hallucination detection.

3. Test-Time Causal View Search (TVS). A three-view cascade (recent, sparse event, compact history) that runs on a frozen ViSS checkpoint, expanding evidence only when a confidence margin is low or when views disagree. It adds +1.56 and +4.56 points on the two benchmarks at a third to a half of the frames of exhaustive three-view selection.

4. A failure attribution via strict replay. By recomputing all three view proposals for every query, the authors attribute 66.9% and 71.9% of the remaining oracle gap to correct answers that were held and then replaced, isolating commitment stability as the main selection limitation within that subset.

Main Findings

• A recent-4 baseline is already strong: Qwen3-VL-8B-Instruct with four recent frames scores 80.52 on StreamingBench and 58.97 on OVO-Bench, above most published systems listed in the paper, including ones that maintain persistent memories or retrieve events.

• Supervision adds on top of generic temporal post-training: With training instances, visual inputs, weighting, ordering, optimization, initialization, and seed matched, generic temporal post-training reaches 84.97 / 63.48 while ViSS reaches 84.77 / 65.21, a +1.73 OVO-Bench advantage and a −0.20 StreamingBench change. Across three paired seeds the OVO gap is always positive (mean +1.72, 95% CI [+0.80, +2.60]).

• The gain is concentrated in hallucination detection: OVO-Bench HLD doubles from 37.63 to 75.27 after ViSS, while EPM moves 54.55 → 57.24 and ASI 62.16 → 60.14. Against the strict-matched generic control, HLD rises from 67.20 to 75.27.

• Sufficiency examples matter: Removing the 116 sufficiency examples lowers OVO-Bench by 1.69 and HLD by 4.30 points, with 124 paired regressions against 67 repairs (McNemar p = 4.5 × 10⁻⁵) and little change on StreamingBench. The authors note this also changes training mass, so it is not an isolated semantic effect.

• The two levers repair different failures: Under fixed recent-4 views, ViSS improves StreamingBench causal reasoning (CR +12.5), action perception (ACP +8.5), and event understanding (EU +5.7). TVS mainly improves prospective reasoning (PR +8.3) and counting (CT +10.9) on StreamingBench, and on OVO-Bench raises EPM to 68.35, ASI to 68.24, REC from 21.20 to 39.83, and CRR to 62.08, while HLD slips to 71.51 and the real-time average falls from 82.20 to 81.27.

• Adaptive access beats fixed and exhaustive selection: With the checkpoint frozen, a single full-history call reaches 86.45 / 68.84 and static 17-frame sampling reaches 85.69. TVS reaches 86.33 / 69.77 at 17.08 vs 29.97 frames on StreamingBench (43% fewer than the history call) and at 30.33 vs 54.33 frames on OVO-Bench, also beating exhaustive max-confidence selection (68.43).

• The acquisition gate, not the commit rule, saves compute: Removing the acquisition gate raises processed frames to 27.26 and 33.31 at essentially unchanged accuracy. Accepting every disagreeing proposal gives 86.53 and 69.75, so the calibrated commit margin is described as an answer-selection heuristic with no consistent aggregate gain.

• Replacement, not availability, dominates the residual gap: A correct alternative-view proposal exists for 43.7% of recent-view errors on StreamingBench and 37.1% on OVO-Bench; TVS returns it in 63.3% and 71.1% of cases and preserves an initially correct recent answer in 96.9% and 92.3%. The three-view oracle reaches 91.42 and 78.52 (5.09 and 8.75 above TVS), and over the 127 and 270 gap queries, 66.9% and 71.9% are replacement errors.

• Overall result: The combined system reaches 86.33 on StreamingBench and 69.77 on OVO-Bench, +5.81 and +10.80 over the same backbone, reported as the best among the published systems compared in the paper. Human reference scores are 91.46 and 92.81.

Methodology in Plain English

The authors treat streaming video answering as two separate design choices and change one at a time on a fixed backbone (Qwen3-VL-8B-Instruct, latest four frames).

Lever 1 — training targets. Instead of only training on question–answer pairs, they build 7,677 training examples that ask the model for the specific temporal facts a bounded view establishes: which events occurred, in what order, how many, which object relations changed, and whether the evidence is sufficient to answer at all. Data comes from source annotations where available (for example, 2,172 repetition-counting probes built on RepCount); where annotations are missing, a multimodal teacher proposes candidates from source frames and timestamps, behind a firewall excluding benchmark media, questions, answers, task metadata, student predictions, and evaluation feedback. Each candidate passes source and timestamp grounding checks plus a "semantic stability" check that keeps it only if the answer maps back to the same target under two option permutations; disagreeing records are removed rather than relabeled. Training is one epoch of masked likelihood on assistant tokens only, taking 98 minutes on two H100 GPUs with a frozen vision tower.

Lever 2 — evidence at inference. The ViSS checkpoint is then frozen and only the view changes. Three question-independent views are compiled from the causal prefix: a recent view (4 frames from the latest 4 seconds), an event view (up to 16 earlier frames ranked by a fused score combining appearance novelty, trajectory curvature, and recency, plus the recent 4, for 20 total), and a compact-history view (8 recent frames plus up to 24 ranked historical frames at reduced resolution, for 32 total). The system starts with the recent view; if the confidence margin—the gap between the two largest full-vocabulary probabilities—is below a threshold it adds the event view, and if recent and event answers disagree it adds compact history. A later proposal replaces the retained one only when its margin exceeds the retained margin by more than a threshold, which is negative, so an accepted proposal need not have a higher raw margin. Both thresholds (τ_a = 0.9976896, τ_c = −0.5059) were calibrated once on 300 queries from 300 source-video-disjoint nonbenchmark videos and then fixed for both benchmarks.

Measurement. The representation effect is the score change at fixed recent-4 access; the access effect is the score change with every learned parameter fixed. Strict replay of all three views on the frozen checkpoint then localizes whether remaining errors are availability failures or selection failures.

Why This Matters

Impact on research: The paper argues that streaming accuracy is not one problem but three—what supervision preserves, what inference exposes, and whether a correct answer survives revision. Its controlled design (matched training budget, frozen-checkpoint comparisons, oracle-conditioned replay) offers a template for attributing gains in streaming video systems, and its negative result on the commit margin challenges the assumption that confidence-based revision rules help.

Real-world applications:

• Wearable or camera-glasses assistants answering questions like "how many push-ups was that?" or "did the garlic go in before the onions?" from partial history.
• Home safety and monitoring queries such as "did I turn the stove off?", where the honest answer may be that the evidence does not suffice.
• Procedural and industrial assistance, where step order and repetition counts matter and a wrong committed answer is costly.
• Efficient always-on video agents that must decide when to spend additional computation rather than processing every frame.

Industry relevance: The frame accounting is the practical hook. TVS matches or beats a full-history call and exhaustive three-view selection at 30.33 versus 54.33 processed frames on OVO-Bench and 17.08 versus 29.97 on StreamingBench, which directly affects serving cost for always-on video inference. The paper also shows that a small, targeted SFT mixture (7,677 examples, one epoch, 98 minutes on two H100s) can move benchmark scores substantially on a strong open backbone.

Future Directions

• Cross-scale and cross-family transfer. The authors state explicitly that the controlled study uses one 8B backbone and two benchmarks, leaving transfer to other model sizes and families open.

• More stable answer commitment. Since 66.9% and 71.9% of the residual three-view oracle gap comes from replacing an already correct answer, the clear next step is a selection rule that preserves correct proposals without suppressing genuine repairs.

• Errors no view resolves. The replay analysis is conditional on proposal coverage; cases where every view fails remain unattributed, and the paper points to at least one example whose ambiguity is what to count.

• Learned rather than fixed retrieval. TVS uses a fixed, query-independent family of three views; the authors name this as a trade-off made to isolate the effect of supervision, leaving learned, query-adaptive retrieval as an open alternative.

Target Audience

Researchers and engineers working on streaming or online video-language models, temporal video understanding, and multimodal post-training, particularly those building always-on video assistants under frame and compute budgets. The controlled-comparison methodology and failure attribution also suit readers interested in rigorous evaluation of training-versus-inference contributions. The paper is less suited to beginners, since much of its argument rests on benchmark taxonomies, margin-based selection, and oracle-coverage analysis.

Authors’ abstract

An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.