OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

AI summary

Overview

Research area: Computer Vision / generative video — specifically reference-to-video (R2V) generation, its evaluation benchmarks, and its training data.

Technical level: Advanced. The paper assumes familiarity with video diffusion models, multimodal reference conditioning, and VLM-based automatic evaluation.

Scope in one sentence: The paper introduces OmniVBench, an 813-case benchmark with 7 task families, 18 fine-grained tasks and 12,172 factor-grounded checklist items, plus the Omni-R2V Dataset of 339,570 (approximately 340K) processed training samples, together forming shared evaluation and training infrastructure for "omni" R2V generation.

Note on completeness: the supplied paper text is truncated mid-appendix (Section A.1.2 and later appendices are cut off), so a few construction and criteria details referenced to the appendix are not available here.

What This Paper Is About

Reference-to-video generation is moving from isolated, single-purpose references toward "omni R2V" — general, compositional control where references can specify content, motion, style, structure, narrative, or combinations of these. Existing benchmarks only cover a subset of this space and mostly score holistic reference similarity, so they cannot tell whether a model preserved the intended factor, disentangled it from the rest of the reference, routed it to the right target, or suppressed it as instructed.

At the same time, building omni R2V training data is expensive, and existing datasets are fragmented across specific reference types. The paper's goal is to supply a shared foundation: a benchmark that evaluates reference factors case by case, and a large-scale, already-processed training dataset covering the same seven task families.

Key Contributions

1. OmniVBench, a broad-coverage R2V benchmark. It spans 7 task families and 18 fine-grained tasks across content, motion, style, structure, narrative, and multi-reference settings, comprising 813 evaluation cases.

2. Factor-grounded evaluation with a validated checklist protocol. Each case is decomposed into case-specific checklist items — 12,172 in total — that assess whether intended reference factors are faithfully preserved, correctly disentangled and bound to their targets, and properly realized per the instruction, over three dimensions: Reference Fidelity (RF), Instruction Realization (IR), and Video Quality (VQ).

3. The Omni-R2V Dataset. A public training resource of 339,570 (approximately 340K) processed samples across 7 task families, built primarily from in-house professional video footage, with clip durations extending to 20 seconds and resolutions from 480p to 2160p+.

4. Task-specific, reusable data construction pipelines for building reference–target pairs, together with an extensive evaluation of 11 open- and closed-source R2V models that exposes gaps across task families and evaluation dimensions.

Main Findings

• Top overall scores on OmniVBench are tightly clustered. Among closed-source models, Seedance 2.5 reaches 72.68 overall, followed by Seedance 2.0 (70.91), Happy Horse 1.0 (70.89), Gemini Omni (70.76), Kling 3.0 Omni (68.59), and Vidu-Q2-Pro (59.45).

• The open/closed gap has narrowed. The strongest open-source model, MiniMax H3, reaches 72.41 overall — comparable to the leading closed-source models. The other open-source results are Bernini 56.95, LoomVideo 51.25, UniVideo 49.69, and OmniWeaving 48.88.

• No model is uniformly strong, and content references are the easiest family. Content scores are generally the highest (Seedance 2.0 at 79.00, Seedance 2.5 at 78.88, MiniMax H3 at 78.86), while larger differences appear on motion, style, structure, narrative, and multi-reference settings — for example Vidu-Q2-Pro scores 71.87 on content but 47.65 on motion, and UniVideo scores 36.97 on narrative.

• Reference Fidelity, Instruction Realization, and Video Quality diverge. The paper reports substantial differences among the three dimensions, showing that strength in one does not imply strength in another.

• The five Reference Fidelity sub-dimensions diverge too. Content, structure, motion, style, and narrative fidelity show large variation, so strong content fidelity does not transfer to motion, structure, or narrative fidelity — the paper's argument for factor-level rather than holistic measurement.

• Disentanglement/routing lags target compliance. For several models, target compliance is relatively high while reference-factor disentanglement and routing (IR_DR) is much lower, especially on multi-content and cross-aspect tasks. One clear case is UniVideo on style: IR_DR 15.39 versus IR_TC 50.78. The paper interprets this as models often following the instruction while copying irrelevant reference content or applying a factor to the wrong target.

• The automatic protocol agrees with human judgment. Across 100 sampled cases (all 7 task families and 18 sub-tasks) producing 965 model outputs rated by three human annotators on a five-point scale, correlation was: RF Pearson 0.81 / Spearman 0.77; IR Pearson 0.78 / Spearman 0.74; VQ Pearson 0.86 / Spearman 0.82.

• The factor-grounded checklist beats holistic scoring. Ablation Spearman correlations: factor-grounded checklist 0.77 (RF) and 0.74 (IR), versus holistic evaluation 0.67 (RF) and 0.69 (IR).

• The dataset is larger and broader than prior R2V datasets in the comparison. Table 2 lists OpenS2V-5M at 5.4M, Phantom-Data at 1M, and MuSS at 30K; Omni-R2V is 340K and is marked as covering both image and video reference modalities, content, motion, style, structure, narrative, multi-content and cross-aspect references, and as directly providing processed data. It is described as non-overlapping with OmniVBench.

Methodology in Plain English

Benchmark design. The authors organize R2V tasks by what the reference controls: content (object, character, scene), motion (action, camera motion such as pan, tilt, dolly, truck, pedestal, orbit, whip-pans, dolly-zooms, and compound movements), style (hand-drawn, painterly, animation, craft, digital, graphic, cinematic), structure (greybox, line art, rough story

Authors’ abstract

Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omni R2V generation. However, existing benchmarks fall short of these emerging capabilities: their test cases cover limited reference types and compositions, and their evaluation protocols largely assess holistic reference consistency, overlooking whether reference factors are properly preserved, disentangled, and routed. Meanwhile, the high cost of constructing omni R2V training data makes suitable training resources scarce. To address these gaps, we introduce OmniVBench and the Omni-R2V Dataset for evaluating and training omni R2V models. OmniVBench expands R2V evaluation across broader reference types, fine-grained control tasks, and richer reference compositions, covering 7 task families and 18 fine-grained tasks spanning content, motion, style, structure, narrative, and multi-reference settings. We introduce factor-grounded evaluation with 12,172 case-specific checklist items, assessing whether intended reference factors are faithfully preserved, correctly disentangled and bound to their targets, and properly realized according to the instruction. We further introduce the Omni-R2V Dataset, bringing industrial-grade training resources for diverse R2V tasks to the broader research community. Drawing primarily on a large-scale corpus of professional video footage, it comprises 340K processed training samples spanning diverse reference types and multi-reference compositions. We develop task-specific pipelines for reference-target pair construction, offering a practical and scalable recipe for omni R2V data construction. Extensive evaluation of advanced open- and closed-source R2V models reveals clear performance gaps across task families and evaluation dimensions on OmniVBench, highlighting remaining limitations of current R2V models.