Transforming Image Editors into Video Editors

AI summary

Overview

Research area: Computer vision and generative AI — specifically instruction-guided video editing built on diffusion models, image editing systems, and image-to-video generation.

Technical level: Advanced. The paper assumes familiarity with diffusion models, VAEs, DiT backbones, ControlNet-style conditioning, and benchmark metrics for video generation.

Scope: The paper proposes Anchor-based Video Editing (AVE), a two-stage framework that turns an existing image editor into a video editor by editing a sparse set of keyframes and then using a frozen image-to-video diffusion model to interpolate the full video around those edited anchors.

What This Paper Is About

Video editing is far harder and more expensive to build than image editing, because video models must handle long spatiotemporal contexts and require scarce, high-quality video editing data. The authors argue that the hard part of video editing is really the underlying 2D image editing problem, and that temporal propagation can be handled cheaply once good edited keyframes exist. Their goal is therefore to transfer the strength of mature image editors into video with minimal additional training, rather than training a monolithic end-to-end video editor.

Key Contributions

1. A decomposition of video editing into two subproblems. The authors frame instruction-guided video editing as (i) composed image editing on a sparse set of keyframes and (ii) temporally coherent interpolation between those edited keyframes, rather than as a single video-to-video task.

2. The AVE framework. A two-stage pipeline in which a powerful image editor edits selected keyframes (Stage-1), and a motion-guided image-to-video diffusion model generates the final video while treating the edited keyframes as fixed anchors (Stage-2). Stage-1 and Stage-2 are fully decoupled, with the edited keyframes as the only interface, so the image editor can be swapped without retraining the video generator.

3. A lightweight motion encoder and anchor-preserving diffusion scheme. The motion encoder is a 6-layer factorized spatial-then-temporal Transformer that extracts motion features from the source video and injects them into a frozen Wan2.2 DiT through a ControlNet-style residual branch. Anchor frames are clamped at every reverse diffusion step and never denoised, and the training loss is computed only on non-anchor frames.

4. Open-source instantiation via multi-image editing. For open-source use, the authors fine-tune BAGEL for consistent multi-keyframe editing, building approximately 1.5M multi-image editing examples from private video editing data.

Main Findings

• Best overall results on IVEBench. On the short subset, AVE reaches a total score of 70.58 versus the previous best 66.75, with video quality 82.10, instruction compliance 49.65, video fidelity 80.01, motion smoothness 89.26, semantic consistency 26.12, and content fidelity 4.10. On the long subset, AVE reaches a total score of 68.75 with quality 82.15, instruction compliance 48.31, fidelity 75.79, motion 88.95, semantic consistency 24.68, and content 4.16. The authors report the best results across the reported metrics on both subsets.

• Motion quality is the standout strength. AVE obtains the best motion score on both IVEBench subsets (89.26 short, 88.95 long), which the authors attribute to Stage-2 treating generation as an anchor-constrained interpolation problem rather than solving semantic editing and temporal reasoning jointly.

• Strong generalization on VIE-Bench across editing types. AVE achieves average scores of 9.054 (add), 9.448 (swap/change), 9.352 (remove), and 9.203 (style/tone), with an averaged instruction-following score of 9.539, preservation score of 9.425, and overall quality of 8.630. It is described as especially strong on swap/change and remove.

• Competitive against closed-source systems. The authors note that the best closed-source models still have a slight edge on a few appearance-centric categories such as add and style/tone, while AVE surpasses them on swap/change, remove, and the averaged instruction-following and preservation metrics.

• Video editing quality tracks image editor quality. In the ablation on IVEBench-long, replacing Nano Banana 2 (default, total 68.75) with GPT-5.1 gives 68.10, Seedream 4.0 gives 67.59, finetuned BAGEL gives 66.07, and untuned base BAGEL gives 58.54. The largest drops appear in instruction compliance, semantic consistency, and content fidelity, while motion stays relatively stable — consistent with the design, since motion is handled by the fixed Stage-2.

• Robustness to keyframe selection. The gap between scene-aware selection (PySceneDetect-based anchors plus largest-gap completion) and uniform sampling is small across budgets. The scene-aware strategy is consistently better, especially with limited budgets. The best budget is approximately ⌊T/24⌋ keyframes (total 68.75 for the default strategy versus 68.21 for uniform sampling at the same budget).

• Too many keyframes can hurt. Performance degrades slightly when keyframes are denser than ⌊T/24⌋ (68.34 at ⌊T/12⌋ for the default strategy). The authors hypothesize that this is because Wan2.2's 3D VAE performs 4× temporal downsampling, so overly dense anchors reduce the effective interpolation space.

• Cheap transfer, not cheap image editing. Stage-2 trains only 710M learnable parameters, about 5% of the diffusion model's parameter count, using 500K video editing samples. The authors report no further improvements when scaling up either data size or training duration.

Methodology in Plain English

The authors avoid training a video editor end to end. Instead, they split the problem in two.

In the first stage, they pick a small number of keyframes from the input video and hand them to a strong image editor together with the text instruction. The important trick is to edit these keyframes jointly rather than one at a time, so the edited frames stay consistent with each other in subject identity, style, and scene layout. By default they use the Nano Banana 2 API (described as Fortin and Alessio, 2026) and edit all keyframes within a single multi-turn conversation. As an open-source alternative, they fine-tune BAGEL on roughly 1.5M multi-image editing examples built by sampling aligned keyframes from paired source and target videos, using a standard diffusion denoising objective on the latent codes of the edited frames.

In the second stage, they use Wan2.2-I2V-A14B as the video generator and keep its 3D VAE and DiT frozen. The only new components are a small motion encoder and a ControlNet-style residual branch. The motion encoder projects the source video's latents into a token sequence and applies alternating spatial and temporal self-attention across 6 layers to extract a motion representation, which is injected into the frozen DiT so that the source video's motion can be preserved.

Generation works by "anchor clamping": the edited keyframes are encoded into latents and pinned in place, while all other frames start from random noise. At every denoising step, the model predicts a noise update, and the anchor frames are immediately overwritten with their clean edited latents again. This guarantees that the edited content at those timestamps is never altered, while the model fills in everything between them using the motion cues. During training, the loss is computed only on non-anchor frames, so the model is explicitly taught to interpolate and propagate between fixed anchors rather than to re-synthesize the anchors themselves.

Training used 64 A100 GPUs, AdamW with a base learning rate of 1×10⁻⁴, weight decay 0.05, β₁ = 0.9 and β₂ = 0.999, a cosine learning rate schedule with linear warmup, and mixed precision.

Why This Matters

Impact on research. The paper reframes video editing as an image editing problem plus a relatively easy temporal propagation problem, and provides evidence for that claim through ablations showing that final video editing quality is strongly correlated with the quality of the Stage-1 image editor. If the claim holds broadly, progress in video editing could come mostly from better image editors plus lightweight transfer, rather than from ever-larger end-to-end video editing training runs. This also suggests a modular research strategy where the two stages advance independently.

Real-world applications:

• Consumer and professional video editing tools that let users apply natural-language edits ("remove the mountains in the background," "turn this into watercolor") while preserving the original motion and camera behavior.
• Object or region removal and replacement in footage, which the paper reports as a particular strength (remove: 9.352, swap/change: 9.448 on VIE-Bench).
• Style and tone transfer across a whole clip, where the paper shows consistent global appearance change while preserving scene composition and wave structure.
• Content production pipelines that need to modify a shot (viewpoint change, object relocation, local attribute editing such as tattoo removal) without reshooting.

Industry relevance. The approach is attractive for industry because it avoids expensive end-to-end video editing training and reuses off-the-shelf image editors, including closed-source APIs. Only about 5% of the diffusion model's parameters (710M) are trained, on 500K samples, with no observed benefit from scaling data or training time. This makes it a practical path for teams that already have a strong image editor and a pretrained image-to-video model. The authors (ByteDance Seed and Johns Hopkins University) note the work was done during an internship at ByteDance Seed.

Future Directions

• Pushing further as image editors improve. Since AVE's video quality tracks Stage-1 quality, the open question is how much of the remaining gap closes automatically as image editors get stronger, and whether the Stage-2 motion encoder ever becomes the bottleneck.
• Understanding the keyframe budget trade-off. The ablation shows a clear optimum around ⌊T/24⌋ keyframes and slight degradation beyond it, hypothesized to stem from the 3D VAE's 4× temporal downsampling. A principled explanation, and selection strategies that adapt per video, remain open.
• Handling large motion changes and failures. The paper shows one example where the couple is edited to start spinning, but it does not report a systematic analysis of when edits require motion changes that go beyond what the motion-guided interpolation can support.
• Removing dependence on closed-source editors. The default configuration relies on the Nano Banana 2 API, and the finetuned open-source BAGEL alternative scores lower (66.07 versus 68.75 on IVEBench-long). Closing that gap with open models is a natural next step.

Target Audience

Researchers and engineers working on generative video, video editing, and diffusion models who want an alternative to end-to-end video editing training; practitioners building editing products who already have a strong image editor and a pretrained image-to-video model; and students or researchers interested in how to decompose a hard spatiotemporal task into modular stages. Readers without background in diffusion models and ControlNet-style conditioning will find the methodology sections dense, though the paper's central idea is conceptually simple.

Note on reporting: the paper does not report inference latency, per-sample cost, the exact size of the IVEBench or VIE-Bench test sets, or a systematic failure-case analysis.

Authors’ abstract

Recent image editing systems have achieved impressive semantic understanding, visual fidelity, and instruction-following ability, while video editing remains substantially more difficult and costly. In this paper, we present a simple alternative to end-to-end video editing: instead of training a monolithic video editor, we transform a strong image editor into a video editor through anchor-based generation. Our key insight is that video editing can be decomposed into two subproblems: editing a sparse set of keyframes and propagating those edits across time. Based on this observation, we propose Anchor-based Video Editing (AVE), a two-stage framework in which a powerful image editor first performs composed editing on selected keyframes, and a motion-guided image-to-video diffusion model then generates the final video by treating the edited keyframes as fixed anchors. This design directly inherits the strengths of modern image editors while avoiding expensive end-to-end video editing training. Experiments on IVEBench and VIE-Bench show that AVE achieves strong performance in instruction following, temporal consistency, and content fidelity. Further ablations reveal that final video editing quality is strongly correlated with the quality of the image editor, suggesting that future progress in video editing may come from stronger image editing foundations and lightweight transfer to video. Code is available at https://github.com/wangf3014/AVE.