VideoCoF: Unified Video Editing with Temporal Reasoner

AI summary

Overview

Research area: Computer Vision — video diffusion models and instruction-guided video editing.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), latent video autoencoders (Video VAE), rotary position embeddings (RoPE), flow-matching / velocity-field training, and in-context learning.

Scope (one sentence): The paper introduces VideoCoF, a Chain-of-Frames video editing framework that forces a video diffusion model to predict an edit region before generating the edited video, achieving state-of-the-art instruction-following on the new VideoCoF-Bench while trained on only 50k video pairs.

What This Paper Is About

Video editing models currently face a trade-off: expert models produce precise, localized edits but depend on external masks, while unified in-context models are mask-free but map instructions to regions poorly. VideoCoF resolves this by making the diffusion model first predict a "reasoning" representation of the edit region and only then generate the edited frames, following a "see → reason → edit" procedure. The goal is precise, mask-free, instruction-guided video editing that also generalizes to video lengths far beyond what the model was trained on.

Key Contributions

1. Chain of Frames (CoF): The first framework to introduce a Chain-of-Frames approach to video editing, enabling temporal reasoning for fine-grained editing by compelling the model to predict the edit-region latents before generating the target video tokens.

2. An effective reasoning format: The authors empirically explore what the reasoning frame should look like and find that a soft, gradually highlighted grayscale (gray, progressively transparent) region is the most effective format—beating black masks and red 50% transparent highlights.

3. A RoPE alignment strategy: By explicitly accounting for the reasoning latent, the temporal indices of the edited video's rotary position embeddings are reset to match the source video, ensuring motion alignment and enabling length extrapolation beyond the training duration.

4. VideoCoF-Bench and data curation: A 200-video benchmark covering four editing tasks with instance-level cases, plus a data curation pipeline (Qwen-VL 72B, Grounding-SAM2, MiniMax-Remover, VACE-14B, GPT-4o, Dover Score, VIE Score) that distills a 50k-pair training set from the Señorita 2M dataset and other sources.

Main Findings

• State-of-the-art instruction following and success ratio: On VideoCoF-Bench, VideoCoF scores 8.97 on GPT-4o Instruct Follow and a 76.36% Success Ratio, the best across all categories. The strongest baseline, ICVE (pretrained on ~1M videos plus 150k for finetuning), reaches 7.79 and 57.76%.

• Large efficiency margin over ICVE: The paper reports a +15.14% improvement in instruction-following score and +18.6% improvement in success ratio over ICVE, achieved with only 50k video pairs versus ICVE's ~1M pretraining videos plus 150k finetuning.

• CoF ablation gains: Comparing the naive temporal in-context baseline to VideoCoF, the instruct follow score increases by 10.65%, the success ratio improves by 5.46%, and CLIP-T rises 4.16%.

• RoPE design improves preservation and structure: The RoPE alignment design yields a 3.4% relative increase in the preservation score and a better DINO score (0.9913 versus 0.9857 for naive temporal in-context), indicating better preservation of spatio-temporal structure.

• Reasoning format matters a lot: A black (background) mask achieves 52.17% success ratio, a red 50% transparent highlight 60.33%, a gray 50% mask 68.45%, and the progressive gray mask (0–75% transparency) 76.36%.

• Four reasoning frames is optimal: Ablating reasoning frame count from 1 to 5 shows 4 frames is best (8.973 instruct follow, 76.36% success ratio). This corresponds to the VideoVAE temporal compression formula L = (F−1)//4 + 1, where frames 1–4 map to a single latent frame and F=5 introduces a second latent frame (which degrades to 7.915 and 29.06%).

• Length extrapolation beyond training: Because of the RoPE alignment design, a model trained on 33 frames extrapolates to 141 frames (4× training length) and 513 frames (16×), with the model stable for the first 500 frames in multi-shot settings and showing slight degradation at 512 frames.

• Competitive on other benchmarks: On TGVE+ (1,417 samples across 7 instruction editing tasks), VideoCoF achieves PickScore 20.90, CLIP-F 0.956, ViCLIP_dir 0.213, ViCLIP_out 0.257, versus EVE's 20.88, 0.926, 0.198, 0.251. On V2VBench, VideoCoF leads on all listed metrics (e.g., Quality 5.024, Motion Align. −2.620 versus MotionDirector's 4.984 and −3.088).

• Triptych prompt is essentially free: Adding the "temporal triptych prompt" raises success ratio from 71.43% to 76.36% and CLIP-T from 27.07 to 28.00, without the expensive instruction-mode pretraining ICVE requires.

• Inference cost: With DMD-LoRA, inference takes only 4 steps and about 10 seconds to edit 33 frames on a single H100 GPU.

Methodology in Plain English

The team fine-tunes a WAN-14B video diffusion model. Instead of concatenating a clean source video with a noised target video (the usual in-context approach) and hoping the model figures out where to edit, they insert a middle segment in the sequence: a "reasoning" clip that visually highlights the region to be edited. At training time the model is given triplets of source, reasoning (a gray, progressively transparent highlight over the edit region), and edited video, and only the reasoning and target parts are noised and supervised; the source latents stay clean.

Three segments are encoded separately by a Video VAE and concatenated along the time axis, then processed in one unified sequence using self-attention for in-context learning and cross-attention for text control. To keep the model from overfitting to a fixed source-to-target position mapping, the authors redesign the rotary position embeddings: source and target videos both use temporal indices [1, F], while the reasoning frame sits at index 0. This gives the reasoning token an isolated slot so it provides spatial guidance without colliding with the first video frames or breaking motion alignment.

Training data was built with a curation pipeline featuring Qwen-VL 72B for multi-instance identification, Grounding-SAM2 for per-instance segmentation, MiniMax-Remover for additions/removals, VACE-14B for swaps and local style transfer with GPT-4o-generated prompts, and Dover Score plus VIE Score for filtering. The final dataset is 50,000 video pairs: 10,000 addition, 15,000 removal (including 5,000 multi-instance), 15,000 swap (including 5,000 multi-instance), and 10,000 local style. Training uses 33-frame videos at four resolution buckets (336×592, 400×704, 400×752, 400×944 and vertical variants) with a 4-frame reasoning clip, for 8k iterations on 16 H100 GPUs.

Why This Matters

The work shows that a video diffusion model can perform explicit visual reasoning about where to edit without any user-provided mask, closing the accuracy gap between mask-dependent expert models and unified in-context models while using roughly an order of magnitude less training data than the strongest baseline.

Real-world applications:

• Video post-production: Removing, adding, or swapping objects and people in footage using natural-language instructions rather than hand-drawn masks.
• Multi-instance editing: Editing a specific instance among several similar objects (e.g., the largest cup among several, or the right person rather than the left), a task the paper shows baselines failing.
• Local style transfer: Applying texture or stylization to a chosen region in a video while leaving the rest untouched.
• Long-form / variable-length content: Because the model extrapolates to 513 frames (16× the 33-frame training length), it suits longer clips without retraining.

Industry relevance: Content creation, advertising, streaming, and social media pipelines stand to benefit from a mask-free, instruction-driven editing model that edits 33 frames in about 10 seconds on a single H100 GPU—low enough latency and hardware cost to be practical rather than purely academic.

Future Directions

• Pushing the extrapolation ceiling: The paper notes that single-shot settings support 16× extrapolation, while multi-shot settings stay stable for the first 500 frames but degrade slightly at 512 frames—an explicit failure case to address.

• Closing the preservation and quality gap: VideoCoF's Preservation and Quality scores (8.20 and 7.77) are slightly below ICVE's (8.06 and 8.14 on Preservation and Quality respectively in Table 1, where ICVE leads on Quality with 8.14), which the authors attribute to ICVE's 1M-pair pretraining and 3× larger supervised fine-tuning set—suggesting scaling data or pretraining as a direction.

• Reasoning-format and reasoning-length search: Only four mask colors/transparency schemes and reasoning frame counts from 1 to 5 were explored; the paper's own finding that a fifth reasoning frame collapses performance (29.06% success ratio) suggests the reasoning representation space is not yet well understood.

• Extension beyond the four tasks: VideoCoF-Bench covers Object Removal, Object Addition, Object Swap, and Local Style Transfer; whether CoF generalizes to other edit types or more complex spatial/physical reasoning is left open.

• Better length-extrapolation methods for V2V: The authors note that T2V extrapolation methods such as RIFLEx and UltraViCo target motion or content repetition rather than source-target alignment and temporal index collision—when applied to the baseline, UltraViCo still fails.

Target Audience

This paper is best suited to computer vision and generative-model researchers working on video diffusion, in-context learning, and diffusion transformer positional encodings; graduate students and engineers building video editing systems; and practitioners in media, VFX, and content creation who need mask-free, instruction-driven editing tools. Readers should already be comfortable with diffusion training objectives and transformer position embeddings.

Authors’ abstract

Existing video editing methods face a critical trade-off: expert models offer precision but rely on task-specific priors like masks, hindering unification; conversely, unified temporal in-context learning models are mask-free but lack explicit spatial cues, leading to weak instruction-to-region mapping and imprecise localization. To resolve this conflict, we propose VideoCoF, a novel Chain-of-Frames approach inspired by Chain-of-Thought reasoning. VideoCoF enforces a ``see, reason, then edit" procedure by compelling the video diffusion model to first predict reasoning tokens (edit-region latents) before generating the target video tokens. This explicit reasoning step removes the need for user-provided masks while achieving precise instruction-to-region alignment and fine-grained video editing. Furthermore, we introduce a RoPE alignment strategy that leverages these reasoning tokens to ensure motion alignment and enable length extrapolation beyond the training duration. We demonstrate that with a minimal data cost of only 50k video pairs, VideoCoF achieves state-of-the-art performance on VideoCoF-Bench, validating the efficiency and effectiveness of our approach. Our code, weight, data are available at https://github.com/knightyxp/VideoCoF.