MoviEdit: Long-form Multi-Shot Video Editing Made Easy

AI summary

Overview

Research area: Computer Vision — generative video editing, specifically long-form, multi-shot video editing with diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching models, DiT backbones, causal autoregressive video generation, attention masking, and KV caching.

Scope: MoviEdit is an end-to-end framework — data pipeline, cascaded bidirectional–autoregressive architecture, and a dedicated benchmark (MoviEdit-Bench) — for editing videos that contain many shots, where a single intended edit must stay consistent across cuts.

What This Paper Is About

Existing video editors are built for short, single-shot clips of a few seconds. When you try to edit a longer video that cuts between scenes, viewpoints, and subjects, the edit tends to drift, land on the wrong subject after a cut, or change appearance each time the subject reappears. The authors argue this is not just a "more frames" problem: there is no training data with cross-shot subject correspondence, naively processing full source plus target sequences is computationally prohibitive, and no benchmark measures cross-shot editing behavior. MoviEdit tackles all three at once.

Key Contributions

1. A scalable, annotation-free paired-data pipeline producing 87K paired multi-shot editing videos (86,619 filtered pairs from 205,228 raw pairs, per Table 3) from three sources: synchronized multi-camera footage, real-world cinematic clips, and a pretrained multi-shot text-to-video generator. It supports pixel-aligned editing, subject localization, and semi-pixel-aligned editing.

2. A cascaded bidirectional–autoregressive editing architecture with shared weights: a lightweight "Draft Editor" produces a globally coherent edit over a spatiotemporally subsampled sequence, and a "Render Editor" autoregressively propagates and refines that draft at full resolution, chunk by chunk.

3. MoviEdit-Bench, a dedicated benchmark of 64 generated multi-shot videos and 128 editing scenarios across eight categories, covering text-guided (V2V) and edited-frame-guided (IV2V) editing, first-frame and arbitrary-frame references, and both local and global edits, with an evaluation protocol spanning seven dimensions.

4. Inference optimizations — unified few-step distillation with asymmetric step schedules (12 steps for the Draft Editor, 4 for the Render Editor) plus KV caching — reported to cut latency from 364.1 s to 11.8 s (a 30.9× speedup) at 540p and 361 frames on 8 H100 GPUs.

Main Findings

• Strongest overall V2V result in the comparison: On MoviEdit-Bench text-guided editing, MoviEdit reaches 0.790 overall edit success vs. 0.764 for Aleph 2.0, 0.623 for HappyHorse-1.0, 0.586 for Gemini Omni Flash, and 0.378 for the best open-source baseline listed (Wan-Edit). It also leads on most individual metrics, e.g. non-subject preservation LPIPS 0.0741 (lower is better), edit-localization precision 0.904, cross-shot edit consistency VLM 0.904, and edit-prompt alignment VLM 0.884.

• Competitive but not uniformly best on IV2V: With a first-frame edited reference, MoviEdit scores 0.803 overall edit success against Aleph 2.0's 0.806; with arbitrary-frame references it scores 0.786 vs. Aleph 2.0's 0.781 (the highest on the full benchmark). The paper frames this as "on par" performance with complementary strengths per metric.

• Cross-shot consistency is where baselines break: Qualitative comparisons report that public baselines propagate artifacts from previous shots, mis-localize edits after cuts, or fail the edit entirely, while proprietary systems like HappyHorse and Gemini Omni tend to edit well but deviate from the source, losing shot transitions and timing and hallucinating motion.

• Draft conditioning suppresses long-horizon drift: A stress test (Fig. 4) shows naive autoregressive editing accumulates appearance drift and oversaturation over long horizons, while draft-conditioned rollout keeps a stable edited appearance.

• Efficiency stack compounds: Starting from 364.1 s (full temporal attention), sliding-window attention gives 258.9 s (1.4×), adding KV caching gives 166.8 s (2.2×), and adding distillation gives 11.8 s (30.9×).

• Scaling helps on both axes: Performance improves consistently when varying training from 0% to 20%, 40%, 80%, and 100% of the 17K-step schedule, and when varying dataset size from 0% to 33% and 100% of the 87K pairs (Fig. 10, with 95% confidence intervals).

• External validation: MoviEdit is also evaluated on the established short-form benchmarks FiVE-Bench and VPBench, with results the authors describe as consistent with MoviEdit-Bench.

• Data filtering is aggressive: Quality control rejects 17%/38% of multi-camera and 32%/47% of cinematic pairs in the two pixel-aligned stages, 24% of localization pairs, and 66% of semi-pixel-aligned pairs.

• Generalization beyond training length: The paper reports that MoviEdit generalizes to videos longer than those seen during training, and also extends to a video set-to-set editing setup.

Methodology in Plain English

Getting training data without humans. The team needed pairs of videos — the same multi-shot footage before and after an edit — with the same subject identifiable across shots. They built three automatic recipes. The main one splits the problem in two: first, lay the first frame of every shot side by side in one image grid and edit them together in a single pass with an off-the-shelf image editor, which forces the edit to look the same across shots; second, propagate each edited frame through its own shot with a single-shot video editor. A second recipe tracks a queried subject through all shots with video segmentation models and applies simple effects (highlighting, outlining, color filling). A third recipe generates source and target pairs from a multi-shot text-to-video model, sharing per-shot captions and initial noise and changing only one global attribute, sometimes sharing the denoising trajectory up to a "fork" timestep so the two videos stay structurally aligned. A vision-language model then filters out bad pairs.

The architecture. Long videos are processed in chunks, which is efficient but gives the model only a short memory — hence drift. MoviEdit's fix is a cheap first pass with global view: the Draft Editor sees a temporally and spatially compressed version of the whole video (M = N/2 in their setting) with full bidirectional attention, producing a coherent skeleton edit. The Render Editor then generates the full-resolution video chunk by chunk, causally, conditioning on recent chunks, the global draft, and the corresponding source chunk. Both stages share the same denoiser weights and differ only in attention pattern and resolution.

Training. The base model is a DiT-based mixture-of-experts backbone with a frozen text encoder and video VAE, initialized from a short-form single-shot editing checkpoint (explicitly with no prior multi-shot ability) and fully fine-tuned on 64 NVIDIA H200 GPUs with FSDP — first 17K steps for V2V, then 8K more for IV2V. The draft stage uses standard flow matching; the render stage uses a causal diffusion forcing objective with independently sampled timesteps per chunk.

Evaluation. MoviEdit-Bench aligns source and edited videos spatially and temporally, computes temporal metrics inside each shot, and combines pixel metrics (LPIPS, NIQE), embedding similarities (CLIP, ViCLIP, DINOv2, PickScore), a motion metric (MFS on tracked points), and a vision-language model judge that gives binary verdicts on subject visibility, edit presence, correct localization, coverage, and preservation of unrelated content. An "overall edit success" score requires every applicable criterion to pass on each sampled frame.

Why This Matters

Research impact. The paper reframes multi-shot editing as a data + modeling + evaluation problem and supplies infrastructure for all three: a supervision recipe that sidesteps manual annotation, a hybrid bidirectional/autoregressive architecture that departs from the current trend of sparse or windowed attention for long context, and the first benchmark the authors identify as targeting cross-shot editing specifically.

Real-world applications:

• Film and episodic post-production: changing a prop, costume, or environment consistently across an entire scene of coverage rather than shot by shot.
• Advertising and branded content: localizing or restyling the same product appearance across many cuts and camera angles.
• Visual effects and clean-up: adding, removing, or swapping elements in long takes and edited sequences while preserving source timing.
• Content repurposing: applying stylization or environment changes to archive or library footage that spans many shots.

Industry relevance. The comparison set includes Gemini Omni Flash, HappyHorse-1.0, Aleph 2.0, GenProp, Lucy Edit, Wan-Edit, and SANA-Streaming, and MoviEdit is reported to match frontier proprietary systems — a claim aimed squarely at production-grade editing tools, with efficiency numbers (11.8 s total at 540p/361 frames on 8 H100s) relevant to deployment cost.

Future Directions

• Beyond 15-second training clips. The model reportedly generalizes to longer videos than it was trained on; how far that extrapolation holds, and whether length-aware training would extend it, is left open.
• Closing the IV2V fidelity gap. On first-frame IV2V, MoviEdit trails Aleph 2.0 slightly (0.803 vs 0.806 overall), and edited-frame fidelity is one of its weaker dimensions (0.797 vs 0.801); improving adherence to an arbitrary user-supplied edit is a clear next target.
• Human evaluation. MoviEdit-Bench relies on pixel, embedding, and VLM-judge metrics; the paper does not report a human study, leaving perceptual validation of the VLM judgments as future work.
• Data-domain expansion. The pipelines depend on synchronized multi-camera footage, cinematic clips, and generative models, with high rejection rates (66% for semi-pixel-aligned pairs); richer or more efficient supervision recipes could raise yield and edit diversity.

Target Audience

Researchers and engineers working on diffusion-based video generation and editing, especially those building long-context or multi-shot systems; teams responsible for production video tooling who need consistent edits across cuts; and benchmark designers interested in how to measure localization and consistency across shots rather than within a single clip. Readers without a background in diffusion architectures will find the problem framing and benchmark design accessible, but the modeling sections require prior knowledge.

Authors’ abstract

Long-form multi-shot video editing requires an intended edit to remain coherent across abrupt changes in scene, viewpoint, and subject appearance, while processing substantially longer visual context than conventional single-shot editors. We introduce MoviEdit, an end-to-end framework for scalable long-form multi-shot video editing that jointly addresses the challenges of data, modeling, and evaluation. To provide cross-shot supervision data at scale, we develop automatic data-generation pipelines spanning synchronized multi-view footage, real-world multi-shot videos, and generated multi-shot content, yielding 87K paired editing videos without manual annotation. Architecturally, MoviEdit employs a cascaded bidirectional-autoregressive design: a lightweight bidirectional stage first establishes a globally coherent editing draft over a spatiotemporally compressed representation, which is then autoregressively propagated and refined over the full video. We further introduce MoviEdit-Bench, a dedicated benchmark for systematic evaluation of long-form multi-shot video editing. Extensive experiments show that MoviEdit substantially outperforms existing open-source models and performs on par with industry frontier models across text-guided and image-guided editing. Visit our project page for complete results and to access the benchmark: https://multi-shot-edit.github.io/