OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing

AI summary

Overview

Research area: Computer vision, specifically generative video models and evaluation benchmark design for instruction-based video editing (IVE).

Technical level: Advanced. The paper assumes familiarity with video diffusion models, flow-matching architectures, vision-language models (VLMs), and multidimensional evaluation metric design.

Scope: The paper introduces OmniEdit-Bench, a 790-sample benchmark spanning five editing tracks (Spatial, Temporal, Audio, Reference, Reasoning) plus a four-dimension VLM-based evaluation pipeline with an accuracy-aware penalty, and uses it to score eight open-source and commercial video editing models.

Paper identity: arXiv:2608.05049v2 [cs.CV], by Chenxuan Miao, Yutong Feng, Yi Lu, Yunfeng Yan, Donglian Qi, Shiwei Zhang, Yu Liu, Xi Chen, and Hengshuang Zhao, with affiliations listed as The University of Hong Kong, Wan Team, Alibaba Group, Zhejiang University, and Peking University. Project page: https://omniedit-bench.github.io.

What This Paper Is About

Instruction-based video editing lets a user change a video by describing the change in natural language, but the authors argue that existing evaluation benchmarks cannot measure this well. Two problems are identified: task coverage is narrow and copied from image editing (mostly frame-level spatial changes, ignoring temporal dynamics, audio, and reference guidance), and the metrics do not actually check whether the instruction was followed, so a model can score highly on a visually pretty video that made the wrong edit. The paper's goal is to build a benchmark and scoring framework that fixes both problems and then to measure how current models perform under that stricter standard.

Key Contributions

1. A five-track, video-specific taxonomy. OmniEdit-Bench organizes editing into Spatial (240 samples), Temporal (200), Reference (200), Audio (100), and Reasoning (50) tracks, explicitly covering dimensions such as motion, sound, and external references that the authors say prior benchmarks do not cover (VE-Bench, EditBoard, OpenVE-3M, IVE-Bench, VIE-Bench, RVE-Bench are compared in Figure 2).

2. A separation of explicit and implicit instructions. Tracks are also organized by instruction complexity, with the Reasoning track requiring models to infer intent rather than follow a directly stated edit. Reasoning is split into physical attribute change, spatial reasoning, temporal reasoning, causal reasoning, and hypothetical reasoning.

3. A four-dimension evaluation pipeline judged by a VLM. Editing quality is decomposed into accuracy, preservation, realism, and consistency, each scored 1–5 by Gemini-3.1-Pro using track-specific prompts.

4. An accuracy-aware penalty mechanism. Rather than averaging the four dimensions, the score is Score = 0.5A + 0.2P' + 0.15R' + 0.15C', where the preservation, realism, and consistency scores are first multiplied by the normalized accuracy  = A/5. Low accuracy therefore suppresses the contribution of every other dimension.

Main Findings

• Overall scores are low for every model. On a 100-point scale, averages range from 14.1 (VIVA, 480P) to 38.3 (KlingV3-Omni, 720P), with Runway Aleph 24.2, Grok Imagine 19.0, Seedance2.0 37.3, Wan2.7-Edit 36.0, Ditto 15.0, and UniVideo 14.4.

• Commercial models lead on spatial editing. The highest spatial score is Wan2.7-Edit at 69.8, followed by KlingV3-Omni 59.6, Seedance2.0 56.9, Runway Aleph 54.2, and Grok Imagine 51.7. Open-source models trail: UniVideo 39.9, VIVA 33.5, Ditto 22.9.

• Temporal editing collapses. Wan2.7-Edit reaches 29.0 and Seedance2.0 25.2, but most other models stay below 20 (Runway Aleph 16.5, Grok Imagine 15.0, KlingV3-Omni 18.5, Ditto 13.5, UniVideo 11.3, VIVA 9.6).

• Reasoning is the weakest language-facing track. The best score is Seedance2.0 at 29.8, with the rest mostly below 25 (Wan2.7-Edit 24.6, KlingV3-Omni 22.9, Runway Aleph 20.3, Grok Imagine 17.9, Ditto 8.6, UniVideo 8.4, VIVA 7.5).

• Audio is underexplored. Only a subset of models supports audio editing at all; reported audio scores are 13.6 for Wan2.7-Edit and 6.7 for Grok Imagine.

• Reference-conditioned generation separates the field sharply. KlingV3-Omni (49.9) and Wan2.7-Edit (46.4) outperform the rest by a large margin, against Runway Aleph 15.1, Grok Imagine 13.1, VIVA 9.8, and UniVideo 4.9.

• Evaluation coverage varies by model for practical reasons. Seedance 2.0 is evaluated only on Spatial, Temporal, and Reasoning tracks due to copyright restrictions, and only models with spatial reference capabilities are evaluated on the Reference track due to architectural limitations.

• VLM judgments track human judgments. Overall mean absolute error between VLM and human scores is 0.86 for accuracy, 0.77 for preservation, 0.55 for realism, and 0.64 for consistency. Samples that received the top VLM score of 5 received human ratings between 4.61 and 4.87, and human score dispersion is lower at higher score levels.

• Scale and data alone do not close the gap. The authors conclude that improvements in model scale and training data primarily benefit spatial tasks but are insufficient for temporal and reasoning challenges.

Methodology in Plain English

The authors first define what video editing tasks exist, then build a dataset of examples, then design a scoring rubric.

For the dataset, they restrict each track to the kind of footage that makes that track meaningful. The Spatial track uses videos with limited motion so consecutive frames look alike, and is split into attribute-level (color, texture, material), subject-level (object removal, replacement, style), and global-level (relighting, weather, season) edits. The Temporal track uses footage with significant camera or object motion and is split into camera attribute, motion attribute, motion semantic, and temporal composition editing. The Audio track covers human speech, object sound, and environmental audio. The Reference track reuses spatial and temporal scenarios but adds a reference image or reference video that the model must align with — reference videos are required for temporal cases because a single image cannot specify motion. The Reasoning track gives implicit instructions, where the desired outcome must be inferred.

Videos come from open-source platforms such as Pexels and Pixabay, public datasets including OpenVid-1M, and outputs of state-of-the-art text-to-video generation models, curated across categories like plants, animals, human activities, natural environments, and human-made scenes, and balanced across domains.

For scoring, each edited video is shown to Gemini-3.1-Pro along with the original video, the instruction, and any reference, and the model returns four 1–5 scores. The prompts instruct the evaluator that accuracy dominates, that visual quality must not be confused with correctness, that preservation should only consider regions unrelated to the instruction, and that a score of 5 should be rare. The three non-accuracy scores are then multiplied by accuracy/5 before being combined with the weights 0.5, 0.2, 0.15, 0.15.

Finally, to check that this automated scoring is trustworthy, the authors grouped samples by VLM score level and compared against human ratings, reporting the proportion of samples at each level, human mean and standard deviation, and mean absolute error per dimension.

Why This Matters

Impact on research. The paper argues that prior IVE benchmarks inherited their task lists from image editing and their metrics do not measure instruction fidelity, which lets models accumulate high scores for edits that were never actually performed. By decomposing evaluation into five tracks and four scored dimensions — and by gating the non-accuracy dimensions on accuracy — OmniEdit-Bench gives the field a stricter and more diagnostic testbed. The human-alignment numbers (overall MAE of 0.86, 0.77, 0.55, and 0.64 across the four dimensions) are presented as evidence that VLM judging is a scalable stand-in for human annotation.

Real-world applications.

• Content creation: natural-language editing that must respect motion, camera, and timing rather than only single frames.
• Film and video production: reference-guided edits where an output must match a supplied style or motion reference over time.
• Education and accessibility: describing a desired change in words rather than operating complex editing software.
• Audio-visual post-production: speech, object sound, and environmental audio edits kept in sync with the picture.

Industry relevance. The benchmark deliberately spans both open-source systems (Señorita-2M, Ditto, VIVA, UniVideo) and commercial ones (Kling Omni / KlingV3-Omni, Seedance2.0, Wan / Wan2.7-Edit, Runway Aleph, Grok Imagine), and its results give a comparable picture of where paid systems currently lead and where nobody performs well. The authors also acknowledge misuse risk from stronger editing tools and position standardized evaluation as a step toward more responsible development.

Future Directions

• Extend temporal and audio coverage. Appendix D states the benchmark is constrained by the scale and diversity of available data, particularly for complex temporal and audio scenarios.
• Reduce reliance on VLM-only judging. The authors note VLM-based evaluation may miss subtle subjective preferences or nuanced quality in borderline cases, suggesting human annotation or hybrid schemes are still needed.
• Broaden the reference and reasoning tracks. The current designs cover only a subset of possible settings, leaving complex cross-modal dependencies and long-horizon reasoning insufficiently explored.
• Improve model capabilities directly. Since temporal, audio, and reasoning scores stay low — for example, the best reasoning score is 29.8 and most temporal scores are below 20 — the authors point to temporal coherence, multimodal alignment, and implicit-instruction understanding as the main open problems.
• Build safeguards. Appendix E and F call for detection and attribution mechanisms and content filtering to mitigate the misuse of improved video editing systems.

Target Audience

Researchers and engineers working on video generation and editing, especially those training or comparing instruction-based video editing models, will get the most from this paper, since it provides both a leaderboard and a scoring rubric they can reuse. Benchmark and evaluation specialists interested in VLM-as-judge reliability will find the human-alignment study relevant. Product and technical leads at companies building commercial or open-source video editing tools can use the track-level breakdown to see which capabilities are still weak. Readers without a background in diffusion or flow-matching video models will need to consult the cited prior work (for example, Ditto, VIVA, UniVideo, Señorita-2M) to follow the related-work discussion, but the benchmark's task taxonomy and scoring formula are presented in accessible terms.

Authors’ abstract

Instruction-based video editing (IVE) is an emerging field with broad applications, yet evaluating editing models remains challenging. Existing benchmarks suffer from two major limitations: limited task coverage inherited from image editing, which overlooks video-specific dimensions, and inadequate metrics that fail to measure instruction fidelity, allowing incorrect edits to receive high scores due to strong visual priors from the original video. To address these issues, we introduce a comprehensive and structured benchmark for IVE. Our benchmark decomposes editing tasks into multiple video-specific dimensions, including spatial, temporal, audio, and reference-based editing, extending beyond conventional frame-level evaluation. It also distinguishes explicit and implicit instructions and incorporates reasoning-based scenarios to better reflect real-world requirements. Furthermore, we propose an evaluation framework that assesses editing quality from four complementary dimensions: accuracy, preservation, realism, and consistency, using both human judgments and state-of-the-art vision-language models. To emphasize instruction fidelity, we introduce an accuracy-aware penalty mechanism that conditions other scores on accuracy, preventing visually plausible but incorrect edits from receiving inflated evaluations. Extensive experiments on representative open-source and commercial models show that current IVE models remain far from satisfactory. OmniEdit-Bench provides a comprehensive and reliable testbed for evaluating instruction-based video editing and offers insights into future research directions.