IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment

AI summary

Overview

Research area: Computer Vision — specifically benchmarking and evaluation of instruction-guided video editing (IVE) systems.

Technical level: Intermediate. The paper is accessible to readers with basic familiarity with generative models, diffusion-based video editing, and evaluation metrics, though it assumes some understanding of multimodal large language models (MLLMs) and video quality assessment.

Scope: IVEBench introduces a large-scale, multi-dimensional benchmark suite for systematically evaluating instruction-guided video editing methods, addressing gaps in source diversity, task coverage, and evaluation rigor present in prior benchmarks.

What This Paper Is About

Instruction-guided video editing — where users edit video content using natural language commands — has advanced rapidly, but existing benchmarks are poorly suited to evaluate it. Prior benchmarks were built for the older "source-target prompt" paradigm, have limited video sources, narrow task variety, and incomplete metrics. This paper introduces IVEBench, a modern benchmark suite with a diverse video corpus, extensive editing task taxonomy, and a three-dimensional evaluation protocol that combines traditional metrics with MLLM-based assessments, validated against human judgment.

Key Contributions

1. A diverse video corpus of 600 high-quality source videos organized across 7 semantic dimensions and 30 fine-grained topics, with lengths ranging from 32 to 1,024 frames. The corpus is split into a short subset (400 videos, 32–128 frames) and a long subset (200 videos, 129–1,024 frames) for long-sequence evaluation.

2. A comprehensive editing task taxonomy covering 8 major categories and 35 subcategories (Style, Subject, Attribute, Subject Motion, Camera Motion, Camera Angle, Quantity, and Visual Effect Editing). Prompts were generated using LLMs (Doubao-1.5-pro) with Qwen2.5-VL-72B captions, then refined through expert review.

3. A three-dimensional evaluation protocol with 12 metrics: Video Quality (SC, BC, TF, MS, VTSS), Instruction Compliance (OSC, PSC, IS, QA), and Video Fidelity (SF, MF, CF). This protocol integrates traditional metrics with MLLM-based semantic assessments.

4. Human alignment validation, demonstrating that IVEBench metrics correlate strongly with human preferences (Spearman's Rho mostly 0.88–0.99) and show substantial inter-rater reliability (Fleiss' Kappa = 0.78).

Main Findings

• Best-performing models still score low overall: Ditto and InsV2V achieved the highest editing capability, but no method exceeded a Total Score of ~4.13 (out of a normalized scale) and Instruction Compliance remained at or below 0.5 across all models — highlighting substantial room for improvement.

• Strong temporal consistency, weak per-frame quality: Frame-to-frame consistency and flickering metrics were generally favorable across methods, but individual frames frequently showed artifacts such as semantic bleeding, boundary blurring, geometric distortion, and texture flickering, degrading Video Fidelity.

• Narrow task versatility: Models performed adequately only on basic editing types (subject, style, attribute). Advanced tasks — quantity editing, subject motion, camera motion/angle, and visual effect editing — were poorly handled by all methods.

• Efficiency varies widely: Lucy-Edit-Dev was the fastest (1.52s per frame on short videos), while AnyV2V and Ditto were slower (11–20s per frame). VACE used the most memory (122 GB) but supported the highest resolution (1280×720).

• Scalability bottleneck for long videos: Most methods exhibit near-linear growth in GPU memory and latency as sequence length increases. InsV2V's chunked inference strategy with latent overlap demonstrated superior scalability for long sequences.

• Resolution ceiling limits practical use: Most models operate at 512×512 or 832×480, far below the 1080p standard users expect, causing blurred textures and edge degradation.

• First-frame-based editing has intrinsic limits: Methods that propagate edits from the first frame struggle with tasks requiring mid-sequence changes such as transitions or inserted events.

Methodology in Plain English

The researchers built IVEBench in three stages.

First, video collection. They manually gathered 600 high-quality videos (≥2K) from Pexels, Mixkit, and open-source datasets, spanning 7 semantic dimensions and 30 topics. Videos went through automated cleanup (removing black borders, subtitles, low-quality content) and manual screening for editability. They then used Qwen2.5-VL-72B to generate structured captions describing subjects, backgrounds, actions, styles, and camera attributes.

Second, prompt generation. For each video, the LLM Doubao-1.5-pro selected an appropriate editing category and generated an edit prompt, along with a target prompt and target phrase for evaluation. All prompts were manually reviewed to ensure balance and clarity across the 35 subcategories.

Third, evaluation design. The team defined a three-dimensional evaluation protocol. They ran 8 state-of-the-art editing models on the benchmark with official implementations, then compared each model's output against human annotators. Thirty participants performed pairwise comparisons guided by trained instructions, and the researchers computed Spearman's rank correlation between automatic metrics and human preferences. They also computed Fleiss' Kappa to confirm annotator agreement, and ran independence tests to confirm the 12 metrics measure distinct qualities.

Why This Matters

Impact on research: IVEBench establishes a shared, human-aligned standard for evaluating instruction-guided video editing, replacing fragmented older benchmarks. By openly releasing all data and code, it enables reproducible comparison and encourages development of methods that address the specific weaknesses it reveals (poor per-frame quality, narrow task support, long-video scalability).

Real-world applications:

• Content creation and post-production: Editors could apply natural-language instructions to modify footage efficiently, provided models support broad enough task coverage and high-enough resolution.
• Film and advertising: Color grading, style transfer, and camera motion adjustments could be automated, though current resolution limitations constrain professional use.
• Personal media applications: Consumers could transform home videos with simple instructions like "add a sunset" or "make this black and white."
• Video restoration and archival: Fidelity-focused editing (preserving unedited content) matters for historical or documentary footage where accuracy is critical.

Industry relevance: The paper provides a clear diagnostic of current commercial and research-grade video editing tools. Findings on speed, memory, and scalability directly inform deployment decisions, while identified weaknesses (instruction compliance, resolution, long-sequence handling) highlight opportunities for product differentiation and further model development.

Future Directions

• Broaden editing task support: Future models need dedicated capabilities for subject motion, camera motion/angle, quantity, and visual effect editing, where current systems perform poorly.
• Improve per-frame quality: Reducing semantic bleeding, boundary blurring, and texture flickering is essential for raising Video Fidelity and moving beyond frame-to-frame consistency alone.
• Solve long-video scalability: More efficient architectures and chunked inference strategies (like InsV2V's) must be developed to make editing hundreds or thousands of frames practical.
• Advance to higher resolution: Closing the gap between typical 512×480 outputs and 1080p+ user content is necessary for professional adoption.
• Expand the benchmark over time: The authors plan to incorporate newly released IVE models and scale up evaluation data as computational resources grow.

Target Audience

This paper is most useful for machine learning researchers and engineers working on video generation, video editing, and multimodal generative models, as well as for benchmark designers interested in human-aligned evaluation methodologies. It also benefits industry practitioners evaluating video editing tools for deployment, and graduate students entering the instruction-guided video editing field who need to understand the current state of the art, its limitations, and the evaluation standards that define the field.

Authors’ abstract

Instruction-guided video editing has emerged as a rapidly advancing research direction, offering new opportunities for intuitive content transformation while also posing significant challenges for systematic evaluation. Existing video editing benchmarks fail to support the evaluation of instruction-guided video editing adequately and further suffer from limited source diversity, narrow task coverage and incomplete evaluation metrics. To address the above limitations, we introduce IVEBench, a modern benchmark suite specifically designed for instruction-guided video editing assessment. IVEBench comprises a diverse database of 600 high-quality source videos, spanning seven semantic dimensions, and covering video lengths ranging from 32 to 1,024 frames. It further includes 8 categories of editing tasks with 35 subcategories, whose prompts are generated and refined through large language models and expert review. Crucially, IVEBench establishes a three-dimensional evaluation protocol encompassing video quality, instruction compliance and video fidelity, integrating both traditional metrics and multimodal large language model-based assessments. Extensive experiments demonstrate the effectiveness of IVEBench in benchmarking state-of-the-art instruction-guided video editing methods, showing its ability to provide comprehensive and human-aligned evaluation outcomes.