Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset

AI summary

Overview

Research area: Computer vision, specifically instruction-based video editing and synthetic training-data generation for generative video models.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, in-context conditioning, model distillation, quantization, and vision-language model agents.

Scope: The paper introduces Ditto (a data synthesis pipeline), Ditto-1M (a one-million-triplet video editing dataset), and Editto (an editing model trained with a modality curriculum learning strategy).

What This Paper Is About

Instruction-based video editing — where a user types a command like "change the suit to black" and the system edits the whole video — has lagged far behind its image counterpart because there is no large, high-quality set of paired training examples (source video, instruction, edited video). The researchers build an automated pipeline that manufactures such data at scale by combining an image editor, a video generator, a depth predictor, and a vision-language model agent, then train a model on the resulting one-million-example dataset. Their goal is a system that edits videos from a text instruction alone at inference time.

Key Contributions

1. Ditto, a scalable synthesis pipeline that generates high-fidelity, temporally coherent video editing data by using an edited key-frame from an image editor as appearance guidance and a depth video as structural guidance for an in-context video generator.
2. Ditto-1M, an open-source dataset of over one million source-instruction-edited video triplets, built with over 12,000 GPU-days of compute.
3. Editto, a state-of-the-art editing model trained on Ditto-1M, which the paper reports outperforms prior methods on automatic metrics and in a human evaluation.
4. A modality curriculum learning (MCL) strategy that anneals and eventually drops the reference-frame conditioning during training, allowing a visually-conditioned model to perform purely text-instruction-driven editing.

Main Findings

• Automatic metrics: On a test set of 50 out-of-distribution videos with 5 instructions each, Editto scores CLIP-T 25.54, CLIP-F 99.03, and VLM 8.10, versus TokenFlow (23.63 / 98.43 / 7.10), InsV2V (22.49 / 97.99 / 6.55), and InsViE (23.56 / 98.78 / 7.35).
• Human evaluation: Based on 1,000 votes from postgraduates and researchers, Editto achieves Edit-Acc 3.85, Temp-Con 3.76, and Overall 3.86, compared with InsViE (2.28 / 2.30 / 2.36), InsV2V (2.17 / 1.96 / 2.07), and TokenFlow (1.70 / 1.97 / 1.70).
• Dataset scale and composition: Starting from over 200k source videos (roughly half featuring human activities), the pipeline yielded approximately 1M edited videos; about 700k are global edits (style, environment) and roughly 300k are local edits (object replacing, adding, removal).
• Dataset format: Final enhanced videos are 1280x720, 101 frames each, at 20 FPS.
• Cost reduction: Combining a distilled video model with a temporal enhancer cuts computational cost to 20% of the high-fidelity baseline, which the paper cites as roughly 50 GPU-minutes per sample on a single GPU.
• Data scaling works: Ablations show that as training sample count increases, both stylistic edit quality and fidelity to the source video's content and motion improve.
• Curriculum learning matters: Without MCL, the model often fails to interpret the full semantic intent of an instruction.
• Model beats its own data generator: Editto substantially outperforms the raw in-context generator used to create the data, particularly on newly emerging content beyond the key frames.
• Synthetic-to-real transfer: Training the model to map the stylized videos back to their real source videos succeeds, indicating the dataset retains photorealistic information.

Methodology in Plain English

The pipeline has a pre-processing stage, a generation stage, and a curation stage.

First, source videos are collected from Pexels under the Pexels License. A visual encoder is used to compute pairwise similarity and remove near-duplicates. To remove low-motion footage, points are sampled on a grid and tracked with CoTracker3; the average cumulative displacement becomes a motion score, and videos below a threshold are dropped. Remaining videos are resized and converted to 20 FPS.

Next, a vision-language model (Qwen2.5 VL) writes a dense caption of each video, then uses that caption plus the video to generate a plausible editing instruction. From the source video, one key frame is selected and edited with the instruction using the image editor Qwen-Image, producing a reference frame that defines the target look. A depth predictor extracts a depth video that encodes the scene's geometry and motion. An in-context video generator (VACE) then synthesizes the edited video conditioned on three things: the text instruction, the edited reference frame, and the depth video.

To keep this affordable, the team applies post-training quantization and uses a distilled version of the generative video model with few-step inference.

Finally, a VLM acts as an automated judge and rejects triplets that fail on instruction fidelity, preservation of the source's semantics and motion, visual quality, or safety and appropriateness. Surviving videos pass through Wan2.2's fine denoiser in a short 4-step reverse process — a small amount of Gaussian noise is added, then removed — to fix subtle artifacts and sharpen texture without changing the meaning of the edit.

For training Editto, the researchers start from VACE and fine-tune only the linear projection layers of the context blocks, keeping most pre-trained parameters frozen. Training runs about 16,000 steps with AdamW at a constant learning rate of 1e-4 on 64 GPUs, using a flow-matching objective. The first 5,000 steps form the curriculum warm-up, during which the edited reference frame is supplied as a visual scaffold; its probability of being provided is then gradually annealed to zero, forcing the model to rely on the text instruction alone.

Why This Matters

The paper's core claim is that the bottleneck in instruction-based video editing is data, not architecture — and it demonstrates that a carefully engineered synthetic pipeline can close the gap that has kept video editing behind image editing. It also provides a concrete recipe for automated quality control at scale using VLM filtering, and shows that a multi-modal training signal can be annealed away so the deployed model needs only text.

Real-world applications:

• Consumer and social video editing: text-driven restyling, object removal, or object replacement without manual masking or rotoscoping.
• Advertising and marketing: rapidly producing stylized variants of the same product footage for different campaigns.
• Film and post-production: propagating a color, style, or wardrobe change decided on a single key frame across an entire shot.
• Content localization: swapping environmental or stylistic elements to fit regional audiences while preserving motion and identity.

Industry relevance is direct: the work comes from Ant Group with academic collaborators at HKUST, Zhejiang University, and Northeastern University, and every component in the pipeline is open-source for reproducibility, which lowers the barrier for others to build similar data engines.

Future Directions

• Extending beyond depth and key-frame priors: the paper notes the original data generator fails on newly emerging information beyond key frames; the trained model does better, but how far this generalizes for long-horizon or heavily occluded edits is open.
• Removing the depth-video dependency: depth is described as a strict spatiotemporal constraint, so testing whether structural context can be learned rather than supplied is a natural next step.
• Scaling the dataset further: the ablations show performance still improves with more samples, raising the question of where the scaling curve saturates and whether the 12,000 GPU-day pipeline can be made cheaper.
• Broader task coverage: Ditto-1M covers global edits (about 700k) and local edits (about 300k); the paper does not report multi-turn or sequential editing, which image editors such as the ones cited already support.

Target Audience

Researchers and engineers working on video generation and editing, particularly those building feed-forward editing models or synthetic data pipelines. It is also relevant to practitioners in generative model training infrastructure, since the paper details distillation, quantization, VLM-based filtering, and curriculum design, and to dataset builders looking for a reproducible open-source template. Readers without background in diffusion or flow-matching models will find the training sections dense.

Authors’ abstract

Instruction-based video editing promises to democratize content creation, yet its progress is severely hampered by the scarcity of large-scale, high-quality training data. We introduce Ditto, a holistic framework designed to tackle this fundamental challenge. At its heart, Ditto features a novel data generation pipeline that fuses the creative diversity of a leading image editor with an in-context video generator, overcoming the limited scope of existing models. To make this process viable, our framework resolves the prohibitive cost-quality trade-off by employing an efficient, distilled model architecture augmented by a temporal enhancer, which simultaneously reduces computational overhead and improves temporal coherence. Finally, to achieve full scalability, this entire pipeline is driven by an intelligent agent that crafts diverse instructions and rigorously filters the output, ensuring quality control at scale. Using this framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of one million high-fidelity video editing examples. We trained our model, Editto, on Ditto-1M with a curriculum learning strategy. The results demonstrate superior instruction-following ability and establish a new state-of-the-art in instruction-based video editing.