TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

AI summary

Overview

Research area: Evaluation methodology for streaming (online) video understanding with multimodal large language models (MLLMs), sitting at the intersection of natural language processing, multimodal video QA, and systems benchmarking.

Technical level: Advanced. The paper is a measurement-protocol and systems-evaluation contribution rather than a new model architecture; it assumes familiarity with causal streaming inference, adaptive benchmarking, and metric design.

Scope in one sentence: TRACE is a condition-aware benchmark and evaluation framework that jointly audits when visual evidence becomes valid, declares the execution conditions under which a system runs, and reports quality, timing, response-selection behavior, workload, completion, and reliability for streaming video understanding.

What This Paper Is About

Current streaming video benchmarks usually report a single task score without saying when the supporting evidence became valid, how the system maintained visual history, who triggered a response, or what processing and failures happened along the way. TRACE argues that this makes scores under-specified: two systems can post similar accuracy while behaving very differently in completion, answer validity, response timing, false alarms, and generation workload. The paper builds a temporally re-audited benchmark plus a unified causal Core–Adapter execution protocol and a multidimensional reporting layer so that a streaming score is always read together with the conditions that produced it.

Key Contributions

1. Condition-aware streaming evaluation. The paper formulates streaming-video evaluation as a system-level measurement problem in which task scores are interpreted under explicit execution conditions. TRACE records visual-state maintenance, response initiation, and evaluation boundary rather than treating these as implicit model properties.
2. Temporally audited tasks and multidimensional measurement. A visual-only evaluation set is constructed from existing streaming-video benchmarks with reviewed evidence timing and instruction-dependent proactive trigger annotations, paired with a unified Core–Adapter protocol that records actual execution behavior and reports quality, timeliness, extra output, workload, completion, and reliability.
3. Empirical analysis of current streaming systems. Eight publicly available models or systems are evaluated in eight configurations, showing that conventional task scores conceal substantial operational differences, and that results obtained with different history-processing mechanisms or evaluation boundaries must be interpreted under their declared execution conditions rather than treated as interchangeable.

Main Findings

• Similar accuracy, different systems. LiveCC and MOSS-Preview reach nearly identical QA Accuracy (65.19% and 65.07%) but differ sharply in operations: MOSS-Preview completes 100.00% of records versus 93.88% for LiveCC, records 42,206 output tokens versus 146,064, and has a lower Invalid Output Rate (1.92% versus 6.12%).
• Proactive performance is not one axis. Proactive results separate into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows, rather than varying along a single common dimension.
• Near-identical proactive accuracy, different behavior. MOSS-VL and AURA have essentially the same In-window Accuracy (8.05% and 7.92%) but differ in Median Response Delay (0.33 s versus 1.15 s) and False-alarm Rate (41.1% versus 59.1%).
• Faster responses and lower miss rates do not imply better selection. MOSS-Preview responds slightly earlier than MOSS-VL at the median (0.20 s versus 0.33 s) and misses fewer target windows (35.67% versus 47.24%), yet has a much higher False-alarm Rate (77.7% versus 41.1%) with lower In-window Accuracy (4.45% versus 8.05%) and more output tokens (193,517 versus 74,672).
• Near-complete coverage can coexist with frequent false alarms. LiveCC has 12.98% In-window Accuracy, misses only 0.31% of target windows, but 65.0% of its assembled response episodes are false alarms; on the shared-video population it exceeds AURA by +4.92 points (95% CI +2.28 to +7.97).
• High off-window output does not guarantee coverage. ThinkStream and VideoLLM-Online combine high Miss Rates (68.43% and 60.39%) with high False-alarm Rates (73.6% and 82.4%).
• Format failure versus capability failure. VideoLLM-Online and MiniCPM-O (native duplex) show low QA Accuracy (2.64% and 2.40%) with very high Invalid Output Rates (95.32% and 93.52%), i.e., outputs not recoverable as a unique option under the text protocol, rather than incomplete execution.
• JoyAI leads the reported proactive quality while being the heaviest observed run. The end-to-end system JoyAI records 17.08% In-window Accuracy, 1.12 s median delay, 50.5% False-alarm Rate, 32.13% Miss Rate, 313,658 submitted images and 797,954 output tokens.
• Temporal audit moves the measurement boundaries. The release ledger contains 328 edited records with 952 field-level changes and 52 excluded records; 325 of the 328 edited records change at least one timing or trigger field (100 of 103 edited QA records and all 225 edited Proactive records). Median absolute revision for directly comparable timestamps is 3.0 s for QA (127 field changes; 90th percentile 58.2 s) and 2.24 s for Proactive (150 field changes; 90th percentile 20.76 s).
• Computation is not a single ranking. Timing and workload measurements remain configuration-specific rather than hardware-normalized, and completion is reported at three granularities with separate denominators (QA record level, Proactive record level, stream level) that are not directly comparable.

Methodology in Plain English

TRACE is built from four connected parts. First, the authors take visual-only tasks from StreamingBench and OVO-Bench — tasks answerable without audio, subtitles, or external knowledge — and convert them into a common structure while preserving source identity. Second, human annotators re-watch each video and mark when the evidence supporting an answer first becomes visible (never later than question arrival), and, for proactive tasks, what the instruction-dependent trigger is: event onset, event completion, the earliest point where clues support a unique answer, or the interval during which a state holds. A combined workflow of manual organization, model-assisted risk screening, and human review corrects or removes records, and the release publishes aggregate review statistics, a per-record change ledger, and an exclusion list.

Third, an Evaluation Core delivers timestamped RGB frames at 1 FPS with real-time pacing and controls when questions or instructions arrive; models can only see video already received. Model-specific Adapters connect this shared input to heterogeneous models and systems while recording actual image submissions, history replay, frame drops, state maintenance, outputs, and failures. The authors explicitly separate Native Streaming (persistent state updated as frames arrive) from Non-native Streaming (a legal causal prefix or window reconstructed at query time), and separately declare whether the measurement covers a model plus its Adapter or a complete end-to-end system with external memory, scheduling, or multiple services.

Fourth, scoring attaches measurements to that execution trace. QA metrics cover Accuracy (correct, uniquely parsed records over all records, with failures, timeouts, and invalid outputs remaining in the denominator), Completion Rate, Response Latency (question arrival to completion receipt, including history replay, input preparation, queueing, and generation), Time to First Token as a diagnostic, Submitted Image Count, Total Output Tokens, and Invalid Output Rate. Proactive metrics use a validity window [t_i, min(t_i + W, t_{i+1})) with the endpoint excluded and W = 5 seconds; responses are assigned to windows by segment onset (first output token) and scored separately for content, so an in-window answer is not necessarily correct. Timeliness is the Median Response Delay, and response-selection behavior is captured by the global False-alarm Rate and the Miss Rate. Proactive content scoring uses normalized exact matching except Sequential Steps Recognition (SSR) and Clues Reveal Responding (CRR), which use a semantic Judge (the same Qwen3.5-35B-A3B Judge configuration across runs). The reported local-model runs used NVIDIA A100 80GB GPUs, and exact per-run checkpoints, Adapter settings, and resolved runtime configurations are released with the project.

Why This Matters

Impact on research. TRACE reframes streaming-video performance as execution-conditioned system behavior rather than a single score. It pushes the field toward declaring evidence boundaries, history mechanisms, response triggers, measurement boundaries, and deployment settings alongside results — and it shows that re-auditing timestamps alone can move the boundaries on which correctness and response timing are computed.

Real-world applications:

• Live monitoring and alerting, where a system must decide when to speak and where false alarms and missed windows carry different costs.
• Interactive assistants and embodied or robotic agents that must act on a stream as evidence arrives, not after the whole video exists.
• Assistive and safety-critical monitoring (for example, industrial or care settings) where knowing whether a system missed a valid window matters as much as its answer quality.
• Deployed video agents built from multiple services, where the evaluated boundary — model plus Adapter, or an end-to-end system with external memory and scheduling — changes how a reported number should be read.

Industry relevance. The benchmark gives engineering teams a way to compare systems on their declared deployment conditions, budget generation volume, track invalid outputs, and separate capability failures from interface/format failures before committing to a streaming deployment.

Future Directions

• Extending the protocol beyond its current scope of visual-only, single-instruction streaming, for example to audio, subtitles, external knowledge, or multiple concurrent instructions.
• Reporting hardware-normalized or otherwise comparable efficiency measures, since TRACE's timing and workload numbers are explicitly configuration-specific and not hardware-neutral rankings.
• Improving sampling and annotation for very short windows: eight Proactive records containing 25 windows of at most one second are released as a sampling-stress subset but excluded from the primary evaluation because they cannot be reliably sampled at 1 FPS.
• Reducing reliance on inherited annotations by moving toward blind duplicate annotation rather than assisted quality control, since annotators in this release could see existing annotations and model opinions.
• Understanding and improving response-selection behavior, given that no single reported quantity — accuracy, completion, answer validity, workload, response timing, false alarms, or misses — reliably predicts the others.

Target Audience

Researchers and engineers who build or evaluate streaming and online video understanding systems, benchmark designers working on temporal annotation and causal-access protocols, and practitioners who need to choose or deploy MLLM-based video agents under real operational constraints such as response latency, generation volume, and false-alarm tolerance.

Note on completeness: the provided paper content is truncated partway through Section 6.2. Findings beyond that point, the remaining analysis, the conclusion, and Appendices A.1–A.3 and B.2 (which hold the wider release landscape, example annotations, Time to First Token, timing coverage, and additional output statistics) are not available in the supplied text, so those details are not summarized here.

Authors’ abstract

Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at \href{https://github.com/om-ai-lab/trace-bench}{https://github.com/om-ai-lab/trace-bench}.