OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing

AI summary

Overview

Research area: Computer vision, specifically instruction-guided video editing (IVE), dataset construction, and diffusion-based generative models.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), vision-language models (MLLMs), mixture-of-experts (MoE) architectures, and standard video-generation training pipelines.

Scope: This paper introduces a 3-million-sample instruction-guided video editing dataset, a 431-pair evaluation benchmark, and a 5B-parameter editing model that outperforms larger open-source baselines.

What This Paper Is About

Instruction-guided image editing has advanced rapidly because large, high-quality open datasets exist to train on. Video editing has not followed suit: existing open video-editing datasets are small, cover few edit types, use very short instructions (often under 5 words), and contain a high proportion of low-quality or failed edits. This paper addresses both problems at once — it builds a much larger and better-filtered dataset, defines a missing standardized benchmark, and demonstrates that training on that data yields a competitive model at a small parameter budget.

Key Contributions

1. OpenVE-3M: a 3-million-sample instruction-guided video editing dataset organized into two top-level categories and eight subcategories. Six are "spatially-aligned" edits that preserve the original motion (Global Style, Background Change, Local Change, Local Remove, Local Add, Subtitles Edit); two are "non-spatially-aligned" edits that keep the subject but change the motion (Camera Multi-Shot Edit, Creative Edit).

2. A reproducible three-stage construction pipeline: preprocessing (video corpus curation, captioning, detection/segmentation, depth and edge maps), taxonomy-guided pair generation (category-specific combinations of image-editing, image-to-video, and video inpainting models), and MLLM-based quality filtering. The pipeline is open-sourced.

3. OpenVE-Bench: a 431-pair, 8-category benchmark with three evaluation dimensions — Instruction Compliance, Consistency & Detail Fidelity, and Visual Quality & Stability — where the Instruction Compliance score acts as an upper bound on the other two.

4. OpenVE-Edit: a 5B-parameter editing model combining an MLLM, a task-aware MoE connector, and a diffusion transformer, which sets a new state of the art among open-source models and surpasses 13B and 14B baselines.

Main Findings

• Scale and instruction richness exceed prior work. OpenVE-3M contains 3M pairs across 8 categories, with an average instruction length of 40.6 words and 65–129 frames at 720p resolution. Comparable open datasets have 3–6 categories, 3.6–17.1 word instructions, and lower frame counts. The category distribution is reported as well-balanced.

• Prior datasets have a bad-case problem, not just a size problem. Sampling 50 edited pairs per category per dataset and scoring them 1–5 with Gemini 2.5 Pro, the authors find that InsViE-1M and Señorita-2M produce a notable fraction of 5-point samples alongside a high proportion of 1-point failures, dragging down the average. Ditto-1M has better quality but concentrates on style transfer.

• OpenVE-Edit beats larger open-source models. With 5B parameters, it achieves the best overall score among open-source methods on OpenVE-Bench, ahead of 13B ICVE and 14B DITTO and VideoCoF. The closed-source Runway Aleph still leads overall (3.50 versus OpenVE-Edit's ~2.41–2.49 depending on the evaluator).

• High temporal quality alone is not enough. OmniVideo scores well on temporal flicker (TF) and inter-frame CLIP consistency (CLIP-F) but essentially ignores the edit instruction, illustrating why the benchmark upper-bounds fidelity metrics by instruction compliance.

• Each architectural component contributes measurably. Using only umT5 text features scores 2.12 overall; adding the MLLM to jointly encode video and instruction raises this to 2.31; replacing a plain MLP connector with the MoE connector raises it further to 2.41.

• Training data quality dominates architecture choices. Training the same model on Señorita-2M yields an overall score of 1.54 versus 2.41 on OpenVE-3M, a gap the authors attribute to quality issues in the comparison dataset.

• Canny edges outperform depth maps as structural control during data generation, particularly when the first and last frames differ significantly; depth control tends to produce poor detail and static backgrounds.

• A 8× generation speedup was needed to make the dataset feasible. Swapping FlashAttention-2 for SageAttention-2 and reducing inference steps from 50 to 10 cut per-video generation from 50 minutes to 6 minutes for an 81-frame 720p clip, with the authors reporting no quality degradation on manual inspection.

• The benchmark aligns with human judgment. Across 300 manually annotated pairs scored by three annotators (inter-rater ICC of 0.948), Seed1.6-VL shows the strongest correlation with human scores (PLCC 0.833, SROCC 0.842), followed by Gemini 2.5 Pro, InternVL3.5-38B, and Qwen3-VL-30B-A3B.

• Quality filtering was done with a mid-tier open model for practical reasons. On a 300-pair human-labeled set, Seed1.6-VL (70%) and Gemini 2.5 Pro (69%) were the most accurate judges, but API rate limits pushed the authors to use InternVL3.5-38B (66%) to score the full dataset, keeping all pairs scoring ≥ 3.

Methodology in Plain English

The researchers start by assembling a base library of about 1 million videos from existing high-quality open video datasets, then cutting each into a 65–129 frame clip at 720p. Every clip is annotated automatically: a large vision-language model writes a long description and lists the objects it can see; detection and segmentation models locate those objects and produce masks; a depth estimator and OpenCV produce structural maps; and a separate model writes short descriptions of each individual object.

With that annotation layer in place, the authors generate editing examples per category, each by a route suited to that edit type. For style transfer, they edit the first frame with an image-editing model, then use a video model conditioned on Canny edges to propagate the style through the clip while preserving motion. For background changes, they first identify the foreground by testing how well object masks overlap a detected foreground mask, then edit only the background and condition the video generation on a foreground-only edge map. For object removal and addition, they run the pipeline in both directions — inpaint an object out of a real video to create an "add this object" training sample, and synthesize an object into a video to create a "remove this object" sample. Subtitles are rendered directly by a tool. The two non-aligned categories use a multi-shot video generator and an image-to-video model to produce creative motion, with shot-detection models splitting multi-shot outputs into edit pairs.

Every generated pair is then scored automatically on the three benchmark dimensions, with instruction compliance capping the other two scores. Because instruction-following is the primary criterion, high-visual-quality videos that ignored the prompt are discarded. Only pairs scoring at least 3 out of 5 are kept.

The model itself, OpenVE-Edit, feeds both the video and the instruction into a vision-language model rather than relying only on a text encoder, so the model captures how the instruction relates to the visual content. A mixture-of-experts connector then routes these features through specialized sub-networks depending on the edit type, using learnable queries so that clips of different lengths and resolutions produce a consistent number of tokens. The final layer of this connector is initialized to zero, so at the start of training it contributes nothing and cannot destabilize the pretrained generation backbone — it gradually learns to contribute as training proceeds.

Why This Matters

Impact on research. Video editing research has been bottlenecked by data rather than architecture. This paper shows that a carefully filtered, instruction-rich, multi-category dataset can let a 5B model beat 14B models, which reframes dataset quality as the primary lever for progress. Releasing both the pipeline and the benchmark also gives the field a shared yardstick, something that image editing already has and video editing has lacked.

Real-world applications.

• Film and television post-production: restyling footage, replacing backgrounds, removing or adding props, and generating shot variations without reshooting.
• Advertising and marketing: producing multiple stylistic or seasonal versions of a single video asset on demand from a text instruction.
• Content localization: adding, removing, or replacing subtitles across positions and styles, a category the dataset explicitly covers.
• Social media and creator tools: converting a single clip into a stylized or creative variant for different platforms and audiences.

Industry relevance. The efficiency results matter commercially. An 8× speedup in data generation, a 5B model that runs at 720p, and a benchmark that correlates with human raters all reduce the cost of building and evaluating video editing products. The finding that dataset quality outweighs model scale also suggests companies with proprietary footage may get more from careful filtering than from larger models.

Future Directions

• Reference-based style transfer is absent. The dataset omits edits that require an external reference image or exemplar, which is a common user request and a gap the authors acknowledge.
• Unifying aligned and non-aligned edits. The current model treats these as distinct regimes. The authors propose adapting image-editing techniques such as noise concatenation along token dimensions to video, but note this remains largely unexplored.
• Scaling model size. Experiments were limited to relatively small architectures by compute constraints, leaving open whether the dataset's gains persist or compound at larger scales.
• Bidirectional synergy between understanding and generation. The authors suggest exploring models where comprehension, generation, and editing capabilities reinforce one another, rather than being trained as separate stages.
• Stronger and cheaper evaluation. Benchmark scoring currently relies on large proprietary VLMs. Whether a smaller open evaluator can match Seed1.6-VL's human alignment (PLCC 0.833) without API constraints is unresolved.

Target Audience

Researchers and engineers working on video generation, video editing, and multimodal diffusion models, particularly those building or fine-tuning instruction-following systems. It is also relevant to dataset and benchmark designers interested in automated quality filtering and VLM-based evaluation, and to applied teams in media, advertising, and post-production evaluating whether open video editing models are ready for production use. Readers without a background in diffusion transformers or vision-language architectures will find the conceptual structure clear but the implementation details dense.

Authors’ abstract

The quality and diversity of instruction-based image editing datasets are continuously increasing, yet large-scale, high-quality datasets for instruction-based video editing remain scarce. To address this gap, we introduce OpenVE-3M, an open-source, large-scale, and high-quality dataset for instruction-based video editing. It comprises two primary categories: spatially-aligned edits (Global Style, Background Change, Local Change, Local Remove, Local Add, and Subtitles Edit) and non-spatially-aligned edits (Camera Multi-Shot Edit and Creative Edit). All edit types are generated via a meticulously designed data pipeline with rigorous quality filtering. OpenVE-3M surpasses existing open-source datasets in terms of scale, diversity of edit types, instruction length, and overall quality. Furthermore, to address the lack of a unified benchmark in the field, we construct OpenVE-Bench, containing 431 video-edit pairs that cover a diverse range of editing tasks with three key metrics highly aligned with human judgment. We present OpenVE-Edit, a 5B model trained on our dataset that demonstrates remarkable efficiency and effectiveness by setting a new state-of-the-art on OpenVE-Bench, outperforming all prior open-source models including a 14B baseline. Project page is at https://lewandofskee.github.io/projects/OpenVE.