StreamEval: Evaluating Interactive Streaming Video Generation Models

AI summary

Overview

Research area: Computer Vision — evaluation of interactive, real-time streaming video generation (avatar/digital-human synthesis, controllable world models, and live video editing).

Technical level: Intermediate. The benchmark design and leaderboard results are readable without deep technical background, though the metric definitions and scoring rules in Section 3.2 assume familiarity with video-generation evaluation practice.

Scope (one sentence): The paper introduces StreamEval, a unified benchmark of 300 scripted, timed interactive video sessions across five tasks and ten metrics, used to evaluate 37 models on how well they execute arriving user requests while preserving prior content and state.

What This Paper Is About

Streaming video models can now generate video incrementally while a user keeps interacting with them, but existing benchmarks mostly judge single clips or single editing turns rather than a whole session of evolving requests. The core problem is that sustained interaction demands two things at once: acting on a new request at the right moment, and not destroying relevant content or state established earlier. StreamEval's goal is to measure both — whether a system does what was asked, when it was asked, and whether it keeps what it was supposed to keep.

Key Contributions

1. A unified interactive benchmark across five streaming tasks. StreamEval covers digital-human dialogue, avatar performance, world generation, instruction editing, and reference editing in one framework, with 300 scripted test cases lasting 15–300 s, each specifying initial assets, incrementally arriving multimodal requests, and explicit change/preserve requirements.

2. A multidimensional evaluation protocol. Ten task-specific metrics are grouped into four dimensions — task execution (Dialogue, Following, Edit Success), state consistency (Stability, Continuity, Preservation), visual fidelity (Quality, Identity, Reference), and audiovisual coordination (A–V Sync) — all on a 0–100 scale with a shared aggregation rule.

3. A cross-task study of 37 models. The authors report 37 distinct models across 47 task configurations on 300 test cases, with three repeats per case and model and equal weighting of the five duration strata.

4. Evidence that execution, not visual quality, is the bottleneck. The reported scores show that high visual fidelity and identity retention can decouple from reliable request following and state preservation.

Main Findings

• Request execution is the central bottleneck. The highest task-specific execution scores reported are 36.90 for Following among seven avatar-performance models (Vidu S2 Avatar), 25.93 for Following in text-conditioned world generation (Krea Realtime), and 15.00 for Edit Success among the three reported reference-editing models (Vidu S2 Editing).

• Dialogue is the most mature task. Vidu S2 Avatar leads digital-human dialogue with 93.98 Overall (88.98 Q-Overall plus the +5 customization bonus), including Quality 96.78, Dialogue 75.29, and Identity 90.12. AVTR-1 is second at 84.14 Overall with 82.97 Quality and 88.54 Identity. Among fixed-avatar systems (Identity rated against each system's own opening frame), Tavus CVI Full reaches 86.65 Q-Overall and 89.89 Identity, versus 65.07 and 87.70 for LiveAvatar FULL.

• High visual quality does not imply good synchronization or conversation. MuseTalk 1.5 records 95.34 Quality, 97.11 Stability, and 88.44 Identity, but only 38.09 A–V Sync and 54.34 Dialogue (79.66 Overall). LiveAvatar FULL reaches 100.00 Stability with 0.00 A–V Sync.

• Scripted actions are rarely executed on time. Across all seven avatar-performance models, Following spans 6.37 (Avatar-Forever 22B) to 36.90 (Vidu S2 Avatar), even where Identity exceeds 80 and Stability exceeds 96. Vidu S2 Avatar (81.71 Overall) and LiveTalk 1.3B (75.85 Overall) lead the task with complementary profiles — LiveTalk has the highest Quality at 71.27.

• World generation splits along quality versus control. MemFlow leads Overall at 33.36, followed by Rolling Forcing at 33.01, Krea Realtime at 31.61, HY-World 1.5 WorldPlay at 31.44, and HappyOyster 1.0 Adventure at 30.84. HappyOyster has the highest Quality (58.34) and Stability (56.67) but low Following (5.56) and Continuity (2.78); Matrix-Game 3.5 has the highest Continuity (34.50); Krea Realtime has the highest Following (25.93), ahead of Reward Forcing (23.33) and MemFlow (22.59).

• Instruction editing is dominated by two systems, and rendering quality stays low. Vidu S2 Editing (45.43 Overall) leads in Following (72.10), Edit Success (57.20), Quality (22.99), and Stability (13.94), with 60.93 Preservation. JoyAI-Video-Edit (42.39 Overall) has the highest Preservation (83.33). SANA-Streaming is third at 16.95; EditaLive (14B) scores 10.29 and XMax X2.0 scores 9.70.

• Reference editing is the weakest task. Vidu S2 Editing leads at 39.02 Overall with Reference 31.20, Preservation 64.44, Stability 46.74, Quality 37.72, and Edit Success 15.00. JoyAI-Video-Edit (14.33 Overall) and XMax X2.0 (9.47 Overall) exceed 20 in Reference but fall to Preservation of 26.67 and 10.00 and Edit Success of 8.89 and 1.11.

• Qualitative case study: models preserve the scene but ignore the reference. In a recorded session where a red velvet cape (Reference A) is preloaded, switched to a gold satin cape (Reference B) at 10 s, and removed at 20 s, both sampled B-phase outputs keep the underlying scene but retain the original grey cape instead of the requested gold satin.

• Benchmark scale and construction. The 300 cases span 240 independent source families plus 60 text-dialogue derivatives, contain 1,347 planned online events, and represent 525 minutes of active streaming generation per pass. Since the paper's benchmark table notes that its 300 s editing cases are a five-fold expansion over LongV2VBench (60 s) and exceed StreamAV-Bench's 180 s world-generation rollouts.

Methodology in Plain English

The authors built a scripted, session-level test suite rather than a collection of one-off prompts. Each test case bundles a starting asset (a portrait, a scene, or a source video), a timeline of user requests that arrive while the session runs, and an explicit statement of what the output should change and what it must preserve. A scheduler releases each input at its designated time, and for editing tasks the source video keeps advancing on the session clock, so a model cannot stall or restart.

Cases were drafted with model assistance: Qwen3.8-Max wrote session goals, timed controls, protected attributes, and expected outcomes, and an authored "oracle" specified what to establish, retain, cancel, or restore at each checkpoint. Media was synthesized (GPT-Image-2 and Qwen-Image for portraits, Qwen3.5-Omni for user and prescribed speech) or drawn from licensed footage including Wikimedia Commons, NASA, and Blender Studio, with per-asset attribution. Every asset passed automated screening: Qwen3-ASR required normalized word-error rates of at most 0.25 against authored transcripts, Qwen3.8-Max audited images for anatomical and rendering defects, and editing sources were checked for scene cuts, frozen spans, and unsuitable targets.

Scoring is done by automated judges under fixed rules, calibrated against acceptance criteria of at most 5% false positives, 10% false negatives, and 10% unresolved decisions per class. Quality rates five-second clips on a 5-to-1 defect scale that maps to 25(r−1); Identity uses two judges (Qwen3.8-Max-0902 and Qwen3-VL-Plus) and takes the lower rating; A–V Sync checks lip-opening/audio-envelope correlation of at least 0.3 at a pooled offset within ±200 ms. Scores are averaged over three repeats, then over test cases within each duration stratum, then equally over the five strata. Missing output and unsupported required operations score zero, and planned but unfired controls stay in the denominator. Avatar tasks add a +5 "sweet-spot" bonus for supporting a custom avatar, so their Overall can reach 105.

Why This Matters

Impact on research. The results reframe the field's priorities: rather than treating photometric quality and identity similarity as the headline metrics, StreamEval shows that following instructions under timing constraints and holding state across a session are where current systems fail. The public leaderboard, code, and dataset (streameval.ai, GitHub, Hugging Face) give the community a shared target, and the finding that rankings reorder depending on which metric you look at argues against single-number model comparisons.

Real-world applications:

• Conversational digital humans for customer service, tutoring, or assistants, where a persona must answer in context and not lose its identity or dialogue history across turns.
• Interactive world models for games and simulation, where a new camera or object command should take effect without scrambling unrelated scene content.
• Live video editing and post-production, where edits must be applied, updated, and then removed to restore the advancing source.
• Accessibility and media production pipelines that need speech-driven avatars to perform requested gaze, expression, pose, or gesture on cue.

Industry relevance. The evaluated roster includes commercial and open systems from many organizations (Vidu S2, Tavus CVI, Runway Characters, LiveAvatar, Krea Realtime, JoyAI-Video-Edit, and others), so the leaderboard functions as a procurement-relevant signal. The finding that a +5 interface-capability bonus and low absolute editing scores coexist indicates that product interfaces are ahead of the underlying controllable generation quality in several areas.

Future Directions

• Improving timed instruction following. With the best Following scores at 36.90 (avatar performance) and 25.93 (world generation), the paper identifies reliable request execution under timing and preservation constraints as the central challenge for omni-modal real-time systems.
• Raising full-edit completion. Edit Success reaches only 15.00 in reference editing and 57.20 in instruction editing, and the recorded cape session shows models preserving the scene while ignoring the reference appearance entirely — raising the question of how to make reference adoption and preservation compatible rather than trade-offs.
• Broadening coverage. The authors note that future extensions can cover broader identities and environments, global resets, and selective restoration while concurrent edits remain active.
• Sustaining rendering quality during live edits. Quality tops out at 22.99 and Stability at 13.94 in instruction editing even for the best system, leaving open how to hold clean temporal rendering while edits are being applied and removed.

Target Audience

Researchers and engineers working on video generation, streaming/causal diffusion models, digital-human avatars, world models, and real-time video editing; benchmark and evaluation researchers interested in session-level, multi-metric protocols; and product or technical-decision teams comparing candidate systems for interactive video applications. Readers new to the area can follow the task descriptions and leaderboard, while the metric definitions and aggregation rules in Section 3.2 will interest specialists.

Authors’ abstract

As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval comprises 300 scripted test cases lasting 15-300s, each specifying initial assets, incrementally arriving multimodal requests, and explicit requirements for state changes and content preservation. Performance is assessed along four evaluation dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination, instantiated through ten task-specific metrics on a 0-100 scale. Evaluating 37 distinct models across 300 test cases, we find that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. In particular, request execution remains challenging: the highest task-specific scores are 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reported reference-editing models. These findings identify reliable request execution under timing and preservation constraints as a central challenge for advancing omni-modal real-time interactive systems. The leaderboard is available at https://streameval.ai