From Draft to Draft-Free: One-Step Video Object Removal via Privileged Distillation and Fast Planting

AI summary

Overview

• Research area: Computer vision, specifically video object removal (VOR) and video inpainting using diffusion models and distillation.
• Technical level: Advanced. The paper assumes familiarity with latent diffusion models, consistency distillation, DDIM sampling, transformer attention masking, and video inpainting benchmarks.
• Scope: The paper introduces D2DF, a three-stage framework that turns a slow, draft-dependent multi-step diffusion refiner into a one-step draft-guided model (D2DF-DG) and then into a fully autonomous one-step model (D2DF-DF) for video object removal.

What This Paper Is About

Video object removal means erasing an object from a video and reconstructing the background behind it. Traditional optical-flow or transformer methods are fast but produce blurry, distorted "drafts," while diffusion-based methods produce realistic results but need many denoising steps, causing hallucinated content and slow inference (tens of DDIM steps). The goal of this paper is to get the realism of diffusion at one-step speed, and then to remove the need for an external draft entirely.

Key Contributions

1. The D2DF framework. A progressive three-stage pipeline that decouples latency from dependency, producing two models: a high-efficiency one-step refiner (D2DF-DG) and a fully autonomous one-step generator (D2DF-DF).
2. Prior-Privileged Consistency Distillation (PPCD). A "truth-injected" distillation scheme in which the teacher is conditioned on the ground-truth video (instead of the flawed draft) during distillation, creating a stable "golden" target trajectory, while the student still takes the draft as input at inference.
3. The Self-Guided Fast Planting (SGFP) module. A lightweight latent-space prior generator built on a Temporal Masked Transformer (TMT) that produces scene-consistent pseudo-drafts, combined with PPCD to create a fully autonomous end-to-end one-step model.
4. State-of-the-art results across three benchmarks (RORD, ROVI, VPLM) plus a zero-shot evaluation on Camera-Bench, with a reported single-video denoising time of about 1 second.

Main Findings

• Stage I teacher (D-LDM): Trained on top of CogVideoX-5B-I2V at 480×720 resolution, conditioned on both the masked video latent and a ProPainter draft, and sampled with 50 DDIM steps.
• D2DF-DG leads most metrics. On RORD: 32.20 PSNR, 0.9318 SSIM, 0.251 VFID, 0.0902 LPIPS. On ROVI: 42.41 PSNR, 0.9888 SSIM, 0.062 VFID, 0.0227 LPIPS. On VPLM: 43.31 PSNR, 0.9877 SSIM, 0.191 VFID, 0.0337 LPIPS.
• D2DF-DF is slightly behind DG but still competitive. RORD: 31.56 PSNR, 0.9202 SSIM, 0.268 VFID, 0.1001 LPIPS. ROVI: 42.25 PSNR, 0.9889 SSIM, 0.063 VFID, 0.0282 LPIPS. VPLM: 43.05 PSNR, 0.9864 SSIM, 0.196 VFID, 0.0348 LPIPS. The paper states D2DF-DF still leads state-of-the-art methods in PSNR.
• Large-mask robustness. Between 0–20% mask ratio, the models show only a marginal PSNR/SSIM advantage over ProPainter and DiffuEraser; the advantage becomes pronounced as mask ratio increases.
• Efficiency. D2DF-DG takes 2.38 s of prior extraction and 1.03 s of denoising, 1 step, for a 3.41 s total. D2DF-DF takes 0.02 s of prior, 1.03 s of denoising, 1 step, for a 1.05 s total. The paper reports D2DF-DF as over 40 times faster than ROSE (45.76 s total, 50 steps), and says complete processing including latent encoding and decoding is within 10 seconds.
• PPCD ablation (RORD). Direct training (D.T.) gives 31.54 PSNR / 0.9192 SSIM / 0.271 VFID / 0.1014 LPIPS; standard consistency distillation (CD) gives 31.85 / 0.9296 / 0.257 / 0.1002; PPCD gives 32.20 / 0.9318 / 0.251 / 0.0902.
• SGFP ablation (RORD). Baseline with no prior: 29.36 PSNR, 0.8920 SSIM, 0.1426 LPIPS. Adding transformer layers: 30.73 / 0.9171 / 0.1149. Adding masked-patch exclusion: 30.78 / 0.9228 / 0.1093. Adding the spatial window: 31.47 / 0.9197 / 0.1070. Adding masked-region-only reconstruction: 31.56 / 0.9202 / 0.1001.
• Three-stage distillation ablation (RORD). D-LDM teacher with CD: 31.3268 PSNR / 0.9179 SSIM / 0.1025 LPIPS. D-LDM teacher with PPCD: 31.4484 / 0.9197 / 0.1014. D2DF-DG teacher with CD: 31.3257 / 0.9173 / 0.1036. D2DF-DG teacher with PPCD: 31.5646 / 0.9202 / 0.1001.
• Zero-shot generalization on Camera-Bench. D2DF-DG with FuseFormer drafts: 27.08 PSNR / 0.9421 SSIM / 0.0831 LPIPS / 0.41 E_warp. D2DF-DG with ProPainter drafts: 26.96 / 0.9429 / 0.0884 / 0.93. D2DF-DF: 26.57 / 0.9365 / 0.0896 / 0.27. E_warp is reported in units of 10⁻⁴.
• Draft-source robustness. Although trained with ProPainter drafts, D2DF-DG successfully refines drafts from DiffuEraser, ProPainter, and FuseFormer, including flow-free FuseFormer drafts.
• Extreme occlusion. On RORD, which uses polygonal mask annotation and contains a high number of extreme occlusions, ProPainter produces blurred textures and MiniMax-Remover (6 steps) shows noise artifacts, while D2DF-DF reconstructs the background in one step.
• Failure cases. When multiple objects with dynamic, complex mutual occlusion relationships are present, both D2DF-DG and D2DF-DF produce imperfect one-step results, though the paper states they retain significantly more context than MiniMax-Remover.
• Stated limitation. Blurring occurs in certain scenarios with one-step denoising, which the authors list as future work.

Methodology in Plain English

The researchers built the system in three stages.

First, they took an existing flow-based method (ProPainter) to produce a "draft"—a rough, often blurry first attempt at the removed video. They then trained a diffusion model (D-LDM), initialized from CogVideoX-5B-I2V, to take that draft plus the masked video and refine it into a high-quality result using 50 DDIM steps.

Second, they compressed that 50-step teacher into a single-step student called D2DF-DG. Standard consistency distillation fails here because the teacher, conditioned on a damaged draft, follows an unstable path. Their fix, PPCD, swaps the draft for the ground-truth video as the teacher's condition during distillation only. This forces the teacher onto a stable trajectory that stays aligned with the ground truth, giving the student a clean target. The student itself still receives the draft as input at inference time.

Third, they removed the draft dependency. The SGFP module is a small Temporal Masked Transformer that works entirely in latent space. It splits the video latent into patches, embeds them, and uses a masked self-attention scheme where any patch can be a Query but only "valid" (mostly unmasked) patches can serve as Keys and Values, optionally restricted to a local spatial window. After 4 transformer layers, the output is un-folded and the unmasked regions are copied back in, yielding a "pseudo-draft." PPCD is then applied again, with D2DF-DG as the teacher, to distill the final draft-free student D2DF-DF, trained jointly with an additional loss constraining the pseudo-draft to the ground truth.

Why This Matters

• Research impact: The paper reframes distillation for video inpainting as a data-conditioning problem—replacing an unreliable condition with a privileged one—rather than purely a step-reduction problem. This "truth-injected" idea may transfer to other conditional video generation tasks where the input prior is noisy.
• Real-world applications:
  • Post-production object removal in film, television, and advertising, where turnaround time matters.
  • Removing watermarks, logos, subtitles, or unwanted bystanders from user-generated and social media video.
  • Anonymizing people, license plates, or sensitive objects in video before public release.
  • Building and cleaning video datasets for training other models.
• Industry relevance: The efficiency numbers are the headline for deployment. D2DF-DF's total of 1.05 s for 25 frames of 480×720 video on an NVIDIA A100, versus 45.76 s for ROSE, moves one-step video object removal closer to interactive and near-real-time editing tools. The paper reports a public code repository at https://github.com/bigD233/D2DF.

Future Directions

• Reducing one-step blurring. The authors explicitly identify blurring in certain one-step denoising scenarios as an enhancement needed in future work.
• Handling complex multi-object occlusion. Failure cases arise when many objects occlude each other dynamically; improving reconstruction of fully occluded content in these cases is an open problem.
• Broader draft-free generalization. D2DF-DF already removes the draft, but the paper does not report results with SGFP on datasets outside the training distribution beyond the reported Camera-Bench zero-shot test.
• Beyond object removal. PPCD and SGFP are presented as general mechanisms for stabilizing one-step conditional video generation, suggesting extension to other video editing tasks. The paper does not report such extensions.

Target Audience

Researchers and engineers working on video generation, video inpainting, and diffusion distillation; practitioners building video editing or post-production tools who care about inference latency; and graduate students studying consistency models and one-step generative methods. Readers need background in diffusion models and transformer attention to follow the methodology and ablation sections.

Authors’ abstract

Video object removal is a fundamental yet challenging task in video editing. Despite recent progress, existing methods typically fall into two categories. Traditional approaches based on optical flow or attention mechanisms often introduce noticeable artifacts and yield unnatural results. In contrast, diffusion-based methods improve visual realism but demand multiple denoising steps, limiting their practicality. To address these issues, we propose From-Draft-to-Draft-Free (D2DF), a framework that distills the ability of transforming coarse drafts into refined videos into a one-step video generation model. Within D2DF, a teacher model is trained to refine low-quality removal results ("drafts") into high-fidelity videos by multiple steps. Then, through Prior-Privileged Consistency Distillation (PPCD), we distill this capability into a student model that performs one-step removal conditioned on the draft. To eliminate draft dependency, we introduce a Self-Guided Fast Planting (SGFP) module based on our Temporal Masked Transformer that autonomously generates scene-consistent pseudo-drafts in latent space, enabling a fully draft-free one-step model. Extensive experiments show that both draft-conditioned and draft-free versions achieve state-of-the-art performance on multiple metrics, surpassing traditional and multi-step generative methods in both quality and efficiency. The denoising process for a single video takes only about 1 second.