Omni-Streaming Thinking

AI summary

Overview

• Research area: Streaming omni-modal (audio + video) large language models, streaming video understanding, multimodal memory and reasoning.
• Technical level: Advanced. The paper is written in formal notation (claims, provenance graphs, verdict-conditioned decoding, preference and on-policy objectives) and assumes familiarity with LLM streaming architectures.
• Scope: The paper diagnoses a specific failure mode in streaming audio–visual reasoning and proposes a claim–verify–retract mechanism, evaluated on five streaming/audio–visual benchmarks plus a new diagnostic benchmark.

What This Paper Is About

Streaming audio–visual models have to decide what to say and when to say it using only the video chunks and synchronized audio observed so far. Because visual cues often "resolve" before an utterance or sound event finishes, a model can lock in an early visual interpretation as a fact and keep repeating it from memory even after the audio contradicts it. The paper names this failure premature cross-modal commitment and proposes Omni-Streaming Thinking (OST), a method that stores unresolved interpretations as future-testable claims, verifies them on the specified modality when the relevant interval closes, and propagates corrections to everything that depended on the original guess.

Key Contributions

1. Diagnosis of premature cross-modal commitment. The authors identify the failure mode in which an interpretation becomes a working fact before the evidence needed to test it has arrived, and argue that streaming memory gives visual bias a lasting effect by carrying early guesses into later decisions.
2. OST, a claim–verify–retract loop. OST generates structured outputs containing evidence observed so far, forecasts of future evidence, and claims built on that evidence. Each claim is marked pending, linked to a future verification interval on a specific modality, and checked when that interval closes; refutations propagate through recorded dependencies to dependent states.
3. OST-DiagBench, a diagnostic benchmark. It holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle–speech conflict, with matched audio-only controls to test whether information missed with video can be recovered from the same waveform alone.
4. Benchmark results. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio–visual benchmarks by more than 10% relative on average, and reaches d′ = 2.95 on OST-DiagBench versus at most 1.38 for open baselines.

Main Findings

• Strongest open result on streaming benchmarks. With audio–visual context, OST's average accuracy on SOVBench-O rises from StreamOV's 81.6 to 87.8, and from 83.8 to 88.4 when prior QA context is also available. The largest gain over StreamOV in the audio–visual setting is on Recall (73.2 to 82.5), the question type requiring earlier evidence to remain usable later.
• Better answer timing. SOVBench-T F1 increases from 90.5 to 92.4, with higher precision and recall for both the Answer and Wait classes.
• Agreement with audio hides visual bias. Audio-blind models produce identical descriptions on Clean and Mute (d′ = 0.00 for Qwen2.5-VL-7B at 73.14/73.14 and Qwen3-VL-30B-A3B at 80.98/80.98). Qwen3-Omni reaches 92.55 Clean source recall yet has a 73.73 Mute hallucination rate; under naive streaming thinking that rate rises to 81.96 while Clean recall stays at 92.55.
• Audio-only recognition can be lost once video is present. Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B have similar audio-only donor recall (77.84 and 79.80), but their Swap attribution scores with video are 60.00 and 35.29. In the Clash condition, the three naive-streaming baselines score 95.69, 80.78, and 94.12 on the edited audio alone, but 0.59, 4.90, and 0.00 when the stale subtitle is visible.
• OST improves recall and correction together. OST has the highest Clean recall (95.49) and lowest Mute hallucination rate (10.39), giving d′ = 2.95. It also leads on Swap attribution (81.76) and Mix joint recall (51.37), and reaches 46.47 Clash accuracy versus at most 4.90 for open baselines.
• Separate evidence retention matters. Removing separated retention (shared audio/video budget) drops SOVBench-O Recall from 82.5 to 74.2 and average accuracy from 87.8 to 84.0; replacing the Omni-State with a free-form scratchpad drops Video-Holmes TCI from 53.1 to 40.7.
• Verification timing matters. Verifying claims at issue time instead of at the specified future window lowers SOVBench-O average accuracy from 87.8 to 83.6; replacing future propositions with non-predictive issues gives 84.1. Removing typed ownership drops d′ from 2.95 to 1.95 and Swap attribution from 81.76 to 63.73.
• Corrections must reach dependent reasoning. Claim-only retraction (no lineage propagation) lowers SOVBench-O Recall to 73.2 and Video-Holmes TCI to 47.6. Setting λ₀ = 0 lowers Video-Holmes average accuracy from 60.0 to 56.2. Removing the hard support gate lowers SOVBench-O average accuracy to 85.2; the latency-matched learned-gate-only no-loop control scores 83.3 versus OST's 87.8.

Methodology in Plain English

OST processes a stream causally. Perception runs every 1 second and decisions are made on 4-second decision chunks; the final chunk may be shorter if the stream terminates.

At each chunk the model writes a six-field Omni-State: visual evidence, audio state (present, absent, or uncertain), audio evidence, conflict, forecast, and sufficiency. The first four fields form the state body.

The model then proposes claims — small records containing a forecast of future evidence, the modality that can verify it (audio, video, or both for relational claims such as source attribution), a future time window in which that evidence should arrive, and a support scope saying whether the claim backs the current state, another claim, or the answer. Each claim starts with a reliability score of 1 and is stored with a unique identifier and its dependencies, so that later reasoning can be traced back to it.

Evidence is kept in separate audio and visual budgets, with pending verification windows pinned until the corresponding claims settle or expire. Long-term memory is a four-way pyramid: 4-second Omni-States are merged into 16-second and 64-second summaries, then into a long-term memory root. Merging carries forward claim identifiers, intervals, verdicts, and reliability scores, so a compressed account of a refuted assumption keeps its reduced reliability.

When a claim's evidence window closes, a verifier checks it against the retained evidence from the specified modality and returns a verdict (Confirmed, Refuted, or Unresolved) plus a contradiction margin. A refuted claim's reliability score is capped, and that reduced score propagates to every descendant through an acyclic provenance graph; during decoding, a log-term added to the attention logits over affected spans reduces their influence. Verdict-conditioned rewriting then contrasts the next-token logits with and without the chunk's refutations, using the difference to steer generation toward the corrected state.

An answer gate decides when to respond. It checks whether the represented evidence is sufficient and whether any answer-critical claims are still awaiting review; OST answers at the first time both conditions hold, or at stream termination. Answers still depending on unresolved claims get a low-confidence flag.

Training uses only unimodal tools plus a schema-constrained Gemini-3.6-Flash annotator to build timestamped, modality-specific evidence records and an offline fact table. With the backbone frozen, the authors train the forecaster, verifier, and gate, then apply supervised fine-tuning to a LoRA policy, and finally run on-policy training through the full loop with rewards for answer accuracy, response timing, and output format. Paired examples that vary only the soundtrack (unchanged, removed, replaced, or with added interference) teach the model to report sounds according to the audio actually present.

Why This Matters

The paper reframes a timing problem in streaming multimodal systems as a memory-integrity problem: it is not enough to perceive audio correctly, because an earlier visual guess can persist and contaminate later answers. Because the method keeps the backbone frozen and adds lightweight adaptation, the approach is attractive for deployment on top of existing omni-modal models rather than requiring retraining from scratch.

Real-world applications:

• Live assistants and broadcast companions that answer questions mid-stream about events, scores, or announcements, where an on-screen graphic may be superseded by a spoken correction.
• Accessibility tools that must report sounds that are present, absent, or replaced, rather than describing what the video merely suggests should be audible.
• Robotics and autonomous systems with synchronized sensors, where a visual reading (e.g., a sign or display) can be overridden by an audio announcement or alarm that completes later.
• Media monitoring and verification, where an edited or dubbed soundtrack conflicts with on-screen subtitles and the system must follow the audio rather than the stale text.

Industry relevance: the benchmarks targeted (streaming, real-time, proactive answering) map onto video platform assistants, meeting and call analytics, and any product that must decide when it has enough evidence to speak. The diagnostic instrument OST-DiagBench is also directly useful for auditing third-party models for visual dominance before shipping them.

Future Directions

• Closing the remaining gap on contradictory speech. Clash accuracy reaches 46.47 versus at most 4.90 for open baselines, but this is still below half; understanding how to fully override a stale on-screen value is unresolved.
• Removing dependence on proprietary annotation. Training data construction relies on a schema-constrained Gemini-3.6-Flash annotator, so open alternatives or self-supervised claim mining would broaden applicability.
• Scaling and cost. Verdict-conditioned decoding requires two forward evaluations per chunk (with and without refutations), and the pyramid uses fixed capacities (6, 4, 4) with a 24-second density floor; how these choices behave on much longer or denser streams is not reported.
• Extending beyond audio and video. The claim schema carries a verifying modality and a support scope, which suggests other sensors or evidence types could be plugged into the same verify-and-retract loop.

Target Audience

Researchers and engineers working on multimodal LLMs, streaming video understanding, and real-time interactive assistants will benefit most, along with benchmark designers interested in diagnosing cross-modal bias. Readers need comfort with probabilistic notation and LLM training pipelines to follow the methods and appendix; the conceptual story about premature commitment and stale memory is accessible to a broader product and evaluation audience.

Authors’ abstract

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.