LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

AI summary

Overview

Research area: Computer Vision / multimodal video-language models, specifically online (streaming) video understanding with Video Large Language Models.

Technical level: Advanced. The paper assumes familiarity with autoregressive transformers, causal attention masking, perplexity-based decoding, and key-value caching.

Scope: The paper proposes LiveStar, a streaming assistant for continuous video streams, together with its training strategy, decoding framework, acceleration techniques, and the OmniStar dataset and benchmark for online video understanding.

What This Paper Is About

Video Large Language Models handle offline video well, but when fed continuous live streams they struggle to decide when to speak, either narrating every frame or falling back on repetitive or "I don't know" outputs. Existing online models lean heavily on predicting an End-Of-Sequence (EOS) token to mark silence, which the authors argue causes response-silence imbalance, inconsistency between adjacent similar frames, misalignment with image-text pretraining, and vocabulary confusion. LiveStar's goal is to decide response timing through verification rather than an EOS token, while preserving the base model's video-language understanding, and to evaluate this on a broader set of real-world scenarios and tasks than prior benchmarks.

Key Contributions

1. LiveStar, a live streaming assistant that performs real-time comprehension of continuous video streams across diverse scenarios and online tasks, responding at contextually appropriate moments.

2. Streaming Causal Attention Masks (SCAM), a streaming video-language alignment training strategy that builds interleaved frame-caption sequences so the model generates temporally consistent captions incrementally over variable-length video prefixes.

3. Streaming Verification Decoding (SVeD), a response-silence decoding framework that determines optimal response timing via a single forward pass verification, plus a peak-end rule-based memory strategy that distills long video contexts by prioritizing salient frames.

4. OmniStar, a dataset and benchmark covering 15 diverse real-world scenarios and 5 evaluation tasks in online video understanding, comprising 20,137 videos.

Main Findings

• Overall online gains: Across all five OmniStar tasks, LiveStar achieves an average 19.5% improvement in semantic correctness (SemCor) with 18.1% reduced timing difference (TimDiff) and a 12.0% increase in FPS compared to the second-best model.

• OmniStar per-task results (SemCor/TimDiff): LiveStar scores 3.19/1.91 on RNG, -/3.57 on OTG, 6.44/1.80 on FDQ, 5.85/- on COQ, and 5.78/- on MIQ, with an FPS of 3.82 on 5-minute videos. For comparison, VideoLLM-online scores 1.68/2.67 (RNG), -/9.69 (OTG), 2.35/2.15 (FDQ), 4.01/- (COQ), 3.83/- (MIQ) at 3.37 FPS; VideoLLM-MoD scores 1.66/2.54, -/9.83, 2.11/2.23, 3.99/-, 3.75/- at 3.41 FPS; MMDuet scores 1.63/2.32, -/4.42, 4.78/2.65, 5.71/-, 5.62/- at 0.91 FPS.

• Offline narration results on Ego4D Narration Stream: LiveStar reaches PPL 1.97, TimeDiff 1.76, and TokAcc 61.1%, compared with LION-FS at 2.09/2.15/52.4% TokAcc, VideoLLM-online at 2.43/2.04/48.1%, VideoLLM-MoD at 2.41/2.04/48.9%, and MMDuet at 4.51/1.97/39.3%. The paper reports this as an 8.7% higher TokAcc than the second-best, LION-FS.

• Offline OmniStar-RNG comparison: LiveStar achieves the best scores among online assistants on PPL (5.14), TokAcc (0.62), SemCor (4.62), and SumFluen (4.55), but remains below human performance (SemCor 6.73, SumFluen 7.17) and below GPT-4V (SemCor 4.97) and GPT-4o (SemCor 5.03).

• Response-silence behavior: VideoLLM-online and VideoLLM-MoD output on nearly every frame, giving the highest timing coverage (TimCover 0.80 and 0.90) but poorer performance on other metrics. MMDuet produces much sparser outputs, yielding the lowest timing redundancy (TimRedun 0.62) at the cost of other scores. LiveStar sits between these, with TimRedun 0.95 and TimCover 0.71 alongside the highest SemCor (3.19).

• Memory dropout ablation: Uniform dropout causes a 4.70% SemCor degradation; FIFO forgetting raises TimDiff by 9.42% with a 3.76% SemCor reduction; peak-end memory compression achieves the best SemCor (3.19) and lowest TimDiff (1.95) without KV cache.

• KV cache ablation: Removing inter/intra-dialogue caching or intra-dialogue caching shows negligible performance loss, while using both boosts FPS by 1.53× and 1.31× respectively on 5-minute video inference. The paper also reports 1.53× faster inference than methods without KV caching with negligible performance loss.

• Threshold sensitivity: The scaling factor α was tested over the interval [1.0, 1.1], with optimal performance in the narrow range α = 1.02 to 1.04; α = 1.03 was selected as the default.

• Caption pool size: M=1 gives SemCor 3.19 / TimDiff 1.91; M=2 gives 3.23 / 1.98; M=3 gives 3.24 / 1.97. M=3 improves SemCor by 1.57% over M=1 but degrades TimDiff by 3.14%, so M=1 is the default.

• Qualitative comparison: Case studies against VideoLLM-online and MMDuet show the baselines suffering from limited contextual understanding, hallucinations, and insufficient fine-grained recognition, while LiveStar gives more accurate, grounded, and timely responses.

• Notable caveat: The paper states that VideoLLM-online did not achieve its reported 10 FPS.

Methodology in Plain English

The authors first change how the model is trained. Instead of teaching it to emit an EOS "stay silent" token for most frames, they build training sequences where each video frame is paired with a caption, and consecutive frames belonging to the same semantic clip share a caption. To prevent the model from cheating by copying a caption it has already produced, they apply a custom attention mask called SCAM that blocks attention to non-terminal captions from earlier clips and to already-generated captions within the same clip. Each turn can only see all video frames from preceding clips and the final caption of each of those clips, which both prevents leakage and lets the model detect scene transitions. Captions are sampled from a pool of paraphrases to reduce overfitting from repeated exposure.

At inference, instead of asking the model whether to emit an EOS token, SVeD computes the perplexity of the caption generated at the last response time. For each new frame, it does a single forward pass to recompute that perplexity. If the new value exceeds α times the reference value (α default 1.03), a decoding gate opens and a fresh caption is generated; otherwise the previous caption is merely moved to the end of the context and no output is produced. This lets the model stay silent until the visual content drifts away from what it last said.

For long videos, the method leans on a "peak-end" intuition from human memory: important moments (keyframes) and recent summaries matter most. Since perplexity is already computed per frame, lower perplexity values are treated as higher semantic importance. Frames older than a prune window W (default 40 frames) are probabilistically pruned, with deletion likelihood tied to relative perplexity within the semantic clip and to elapsed time. A dual-level streaming key-value cache stores both intra-dialogue and inter-dialogue representations so historical context is not recomputed, and it is designed to handle the cache-sequence swaps that SVeD introduces.

The model itself is built on InternVideo2.5, using InternViT as the vision encoder (frames at 1–4 FPS, 16 tokens per frame), an MLP projector, and InternLM2.5-7B as the language model. Training used full fine-tuning on 8× NVIDIA A800 GPUs over 83K data for 1 epoch, learning rate 4×10⁻⁵, AdamW (β₁=0.9, β₂=0.999, weight decay 0.05), per-device batch size 1 with gradient accumulation over 4 steps (effective batch size 32), cosine scheduling with 0.03 warmup, frames resized to 448×448, patch downsampling ratio 0.5, and an 8,192-token training sequence limit within an 8K-token context window. The vision encoder was frozen; the projector and language model were fully updated. Loss was computed only on assistant response tokens.

The OmniStar dataset was assembled with a semi-automated temporally dense annotation pipeline. It contains 20,137 videos spanning 15 scenarios subdivided into 46 categories using YouTube's native annotation system, split into 19,137 training and 1,000 evaluation instances (200 per task) without overlap. Its five tasks are Real-time narration generation (RNG), Online temporal grounding (OTG), Frame-level dense QA (FDQ), Contextual online QA (COQ), and Multi-turn interactive QA (MIQ). Captions are written as narrative-consistent sequences rather than disjointed descriptions, and QA supervision comes in two forms: temporally distributed multi-turn QA chains, and frame-level streaming QA where one standing query has time-varying answers.

Why This Matters

Impact on research: The paper reframes the online video understanding problem away from EOS-token prediction toward verification-based response timing, and provides a training mask (SCAM) plus a decoding scheme (SVeD) that can be applied to other streaming Video-LLMs. It also broadens evaluation from single-task streaming QA on mostly first-person data (Ego4D) toward 15 scenarios and 5 tasks, addressing what the authors call narrow scenario coverage, single-task evaluation, and offline assessment of temporal reasoning and response timing in prior benchmarks such as SVBench, OVO-Bench, and StreamBench.

Real-world applications (as described in the paper):

• Live streaming platforms, where an assistant comments on or narrates ongoing content.
• Surveillance systems, where continuous streams need event-triggered rather than constant output.
• Cinematic tools and real-time interactive editing, requiring temporal grounding in continuous streams.
• First-person and wearable camera devices, such as the egocentric narration setting evaluated on Ego4D.

Industry relevance: Three co-authors are affiliated with Kuaishou Technology, a live streaming and short video platform, and the model is designed to process 10+ minute video streams at 3 FPS, which is the regime streaming platforms actually operate in. The KV cache and peak-end pruning work target deployment latency rather than leaderboard scores alone. The model and dataset are released at https://github.com/yzy-bupt/LiveStar.

Future Directions

• Closing the gap to human performance and to closed-source LVLMs: on OmniStar-RNG offline evaluation, humans score SemCor 6.73 and SumFluen 7.17 versus LiveStar's 4.62 and 4.55, and GPT-4V scores 4.97 SemCor.
• Improving online temporal grounding and real-time QA, where OTG is the weakest reported result (TimDiff 3.57 against human 1.81).
• Extending beyond the current scale of videos and tasks, since the paper frames OmniStar as an initial answer to limited scenario coverage, and the 15 scenarios are drawn from YouTube categories.
• Resolving the trade-off the paper documents between semantic richness and temporal consistency in the paraphrased caption pool (larger M improves SemCor but degrades TimDiff), and further tuning the α threshold range [1.0, 1.1].

The provided paper content is truncated during the Limitations section (it cuts off mid-sentence), so the authors' full discussion of limitations is not reported here.

Target Audience

Researchers and engineers working on Video-LLMs, streaming or online multimodal inference, and real-time human-computer interaction; practitioners building assistants for live streaming, surveillance, or wearable video; and benchmark designers interested in temporally dense annotation and timing-aware evaluation metrics such as TimDiff, TimRedun, TimCover, SemCor, and SumFluen. Readers need prior familiarity with transformer attention masking and decoding to follow the SCAM and SVeD sections in detail.

Authors’ abstract

Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous frame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative coherence. To address these limitations, we introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53x faster inference. We also construct an OmniStar dataset, a comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evaluation tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-art performance, achieving an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0% across all five OmniStar tasks. Our model and dataset can be accessed at https://github.com/yzy-bupt/LiveStar.