LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding

AI summary

Overview

• Research area: Computer Vision / multimodal large language model (MLLM) evaluation, specifically video understanding benchmarks for interactive livestream content.
• Technical level: Advanced.
• Scope: The paper introduces LiViBench, described as the first omnimodal benchmark for interactive livestream video understanding (video, audio, speech, and real-time comments), and presents LiVi-LLM-7B, a 7B-parameter model tuned for that domain.

What This Paper Is About

Existing video benchmarks mainly cover non-interactive content such as movies, recordings, and short videos, so it is unclear how well multimodal models understand livestreams, where streamers and audiences interact in real time through gifts, conversation, comments, and multi-person co-streaming. The paper builds an evaluation benchmark covering these interactive features and then builds a model that performs better on it.

LiViBench contains 3,168 livestream videos (durations from 14 seconds to 33 minutes) and 3,175 high-quality multiple-choice questions spanning 24 tasks in 9 vertical livestream domains. The authors also train LiVi-LLM-7B using 37,953 machine-annotated and 11,180 manually annotated samples, plus a Video-to-Comment Retrieval (VCR) module.

Key Contributions

1. First omnimodal interactive livestream benchmark. LiViBench covers audio, speech, and comment modalities, with 24 tasks grouped into five categories: 4 coarse-grained perception, 6 fine-grained perception, 3 knowledge-based reasoning, 4 general reasoning, and 7 livestream-specific tasks.
2. A standardized semi-automatic annotation workflow with human-in-the-loop at multiple stages. It uses a multi-agent MLLM system (LLaVA-Video, Qwen2.5-VL, InternVL3, and Seed1.5-VL as different "expert" roles) for video description, and a seed-question-driven method for question construction, followed by human review.
3. A tailored two-stage instruction-tuning strategy and a Video-to-Comment Retrieval (VCR) module, used to build LiVi-LLM-7B, a model with enhanced interactive-livestream knowledge.
4. An empirical demonstration that the resulting 7B model outperforms much larger open-source models (up to 72B parameters) on LiViBench and generalizes to general video benchmarks.

Main Findings

• General perception is easier than interaction. Coarse-grained and fine-grained general perception tasks consistently score higher than other task types for both proprietary and open-source models, while the livestream-specific category is the most challenging for all models.
• Proprietary models struggle on LiViBench. GPT-4o scores 56.3 overall and Gemini 2.5 Pro scores 56.1, while Gemini 2.5 Flash scores 53.0. Seed1.5-VL attains the highest proprietary overall score at 66.2, and Doubao-Seed-1.6 reaches 64.9.
• Best proprietary category performance is split. Doubao-Seed-1.6 performs best on Coarse (72.9) and Fine (73.2) tasks, while Seed1.5-VL leads knowledge (68.8), reasoning (70.7), and livestream-specific (59.1) tasks.
• The 7B model beats larger open-source models. LiVi-LLM-7B achieves an overall accuracy of 64.4, outperforming Qwen2.5-VL-72B (62.3), LLaVA-Video-72B (60.0), and InternVL3-78B (64.4, which it matches), and surpassing proprietary GPT-4o (56.3) and Gemini 2.5 Pro (56.1). Its livestream-specific score of 60.9 is the highest in the table.
• Audio helps interactive understanding. In the audio/speech analysis, LiVi-LLM-7B improves from 61.4 (video only) to 63.9 with audio and 63.4 with speech (ASR). All three omnimodal models tested (MiniCPM-o-26, Qwen2.5-Omni, LiVi-LLM-7B) show significant improvement in the livestream-specific category, suggesting audio is important for interactive video.
• Speech is useful but noisier than audio. All models improve in most categories with ASR, but the paper reports that speech can sometimes degrade fine-grained and reasoning performance. Audio is described as more helpful than speech overall.
• Raw comments hurt most models; retrieval helps. Using raw comments degrades most models; InternVL3 is an exception because it uses far fewer frames and therefore has more context space. The VCR module brings LiVi-LLM-7B from 63.9 (video) to 64.4 overall, and from 55.3 (raw comments) to 63.0 in the [1k, ∞) comment-count bucket.
• Comment volume matters. Comments are bucketed into [0, 20), [20, 100), [100, 1k), and [1k, ∞); LiVi-LLM-7B scores 63.7, 64.2, 64.5, and 63.8 with video only, versus 63.8, 65.6, 66.1, and 63.0 with VCR.
• Training-stage ablation. Starting from a baseline of 60.3 on LiViBench, using only the authors' synthetic data in stage one and manual data in stage two yields 63.9 on LiViBench and Video-MME scores of 79.6 / 69.4 / 56.6 (Short / Med / Long). Adding LLaVA-Video-178k general data in stage one (LV+Ours) improves Video-MME (80.6 / 69.4 / 57.3) but lowers LiViBench to 63.1, indicating a trade-off between interactive and general understanding.
• Generalization to general benchmarks. On Video-MME the model scores 79.6 (Short), 69.4 (Med), 56.6 (Long), 68.5 overall; on MLVU it records 70.5 (M-Avg); on LongVideoBench 59.6; and on VideoEval-Pro 50.5 (MCQ). The paper states it achieves the best scores on all tasks of Video-MME and VideoEval-Pro among models of similar parameter size.
• Dataset scale details. Filtering over 30,000 videos left 5,245 videos (20 seconds to 60 minutes) as the benchmark basis; the final benchmark has 3,168 videos, each with audio, comments, and ASR. The dataset contains approximately 1.45 million comments with an average length of 12.15 Chinese characters. Word clouds show terms like "performance," "action," and "interaction" in question sets and "anchor" and "audience" in option sets, compared against the general benchmarks VideoEspresso and MVBench.
• Not reported. The paper does not report annotation cost figures, per-task per-model breakdowns beyond the category averages shown in the tables, or training hyperparameters (learning rates, batch sizes, compute used).

Methodology in Plain English

The authors started by collecting publicly accessible livestream videos with synchronized video, audio, speech, and comment streams. To avoid easy or trivial content, they used a proprietary model (Seed1.5-VL) to score each video's spatiotemporal complexity from 1 to 10 and removed everything scoring below 3, along with videos focused on web games or e-commerce.

Instead of relying on a single model to describe videos (which imports that model's biases) or on pure manual annotation (which is expensive), they built a multi-agent system in which different MLLMs play different expert roles: one handles scene and style, one handles fine-grained detail, one handles logical event flow, and one handles cultural and trend knowledge. The combined descriptions feed a "seed question" library — question templates first generated by a proprietary model and then reviewed and revised by people. A proprietary model (Doubao-Seed-1.6) generates candidate questions for each video from these seeds, and human annotators filter and refine the questions, generate answers, and vet distractors.

For the model, they take Qwen2.5-Omni as the starting point, encode video frames with Qwen2.5-VL's visual encoder and audio with Qwen2-Audio, and fuse the two streams with a transformer decoder before feeding them to the language model. Training happens in two stages: first alignment on 37,953 synthetic samples, then fine-grained tuning on 11,180 human-refined samples, with optional general video data mixed in. At inference, because livestreams produce far too many comments to fit in context, the VCR module uses Chinese-CLIP to embed sampled frames, embeds all comments with a text encoder, matches frames to comments by similarity to pick the top-k relevant comments per frame, and feeds those comments in chronological order with the question.

Why This Matters

• Research impact: The paper identifies a coverage gap in video understanding evaluation — interactive livestream content — and provides a benchmark plus a baseline model, giving the field a measurable target for interactive video understanding rather than only movies, recordings, and short videos.
• Real-world applications:
  • Content moderation on livestream platforms, where models must interpret streamer behavior, audience reactions, and gifting in real time.
  • Automated highlight or clip generation from long livestreams, requiring event and interaction recognition.
  • Livestream recommendation and search, which depend on understanding a stream's topic, style, and audience engagement.
  • Accessibility and analytics tools that summarize what happened in a stream, including speech and comment threads.
• Industry relevance: The work is a collaboration between Peking University and Douyin Group, and it benchmarks Douyin-adjacent models (Seed1.5-VL, Doubao-Seed-1.6) alongside Qwen, InternVL, Gemini, and GPT-4o, reflecting direct platform interest in livestream comprehension. The finding that a 7B model can match or beat models up to 72B parameters on this domain suggests specialized tuning and comment retrieval are practically valuable for deployment at scale.

Future Directions

• Closing the livestream-specific gap. The livestream-specific task category is the weakest for every model evaluated, including the best proprietary systems, so improving interaction-level reasoning remains open.
• Better use of speech and comments. Speech sometimes hurts fine-grained and reasoning performance, and raw comments degrade most models, leaving room for stronger noise filtering and retrieval than the VCR module provides.
• Balancing domain specialization and general capability. The ablation shows a trade-off between LiViBench performance and Video-MME performance when general data is added, so how to get both simultaneously is unresolved.
• Extending and diversifying the benchmark. The authors present LiViBench as a foundation for future interactive video understanding research; scaling tasks, domains, and modalities beyond the 9 vertical domains and 24 tasks covered here is a natural next step.

Target Audience

This paper is best suited for researchers and engineers working on multimodal large language models, video understanding benchmarks, and multimodal data annotation pipelines, as well as practitioners at livestream and short-video platforms who need to evaluate or build models for interactive video content. Readers interested in human-in-the-loop data construction and in efficient domain adaptation of small (7B) models will also find it relevant.

Authors’ abstract

The development of multimodal large language models (MLLMs) has advanced general video understanding. However, existing video evaluation benchmarks primarily focus on non-interactive videos, such as movies and recordings. To fill this gap, this paper proposes the first omnimodal benchmark for interactive livestream videos, LiViBench. It features a diverse set of 24 tasks, highlighting the perceptual, reasoning, and livestream-specific challenges. To efficiently construct the dataset, we design a standardized semi-automatic annotation workflow that incorporates the human-in-the-loop at multiple stages. The workflow leverages multiple MLLMs to form a multi-agent system for comprehensive video description and uses a seed-question-driven method to construct high-quality annotations. All interactive videos in the benchmark include audio, speech, and real-time comments modalities. To enhance models' understanding of interactive videos, we design tailored two-stage instruction-tuning and propose a Video-to-Comment Retrieval (VCR) module to improve the model's ability to utilize real-time comments. Based on these advancements, we develop LiVi-LLM-7B, an MLLM with enhanced knowledge of interactive livestreams. Experiments show that our model outperforms larger open-source models with up to 72B parameters, narrows the gap with leading proprietary models on LiViBench, and achieves enhanced performance on general video benchmarks, including VideoMME, LongVideoBench, MLVU, and VideoEval-Pro.