Object-WIPER : Training-Free Object and Associated Effect Removal in Videos

AI summary

Overview

• Research area: Computer vision — training-free video object removal and inpainting with pre-trained text-to-video diffusion transformers (DiTs).
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, MMDiT joint attention, inversion/denoising samplers (RF-Solver), and self-/cross-attention analysis.
• Scope in one sentence: The paper presents Object-WIPER, a training-free pipeline that removes an object together with its associated visual effects (shadows, reflections, mirrors, translucent artifacts) from real videos, and introduces a new evaluation metric (TokSim) and a new real-world benchmark (WIPER-Bench) to measure that capability.

What This Paper Is About

Existing video inpainting and object removal methods are given only a mask of the object and fill just that region, so an object's associated effects — its shadow, its reflection in water, its mirror image, the light passing through a translucent body — remain in the output video and create obvious artifacts. Training-free editing approaches that try to handle these effects either seed their effect masks from the user's object mask alone (missing weakly activated regions) or depend on an external point-tracking model whose errors propagate into the attention computation.

Object-WIPER instead asks a pre-trained text-to-video diffusion transformer directly, using text describing the object and its effect, to localize both the object and its associated effect, then removes and inpaints them while preserving the background — all without any fine-tuning or extra training data.

Key Contributions

1. A training-free pipeline (Object-WIPER) that localizes an object's associated effects inside MMDiT blocks by combining text-to-visual cross-attention with visual self-attention, then removes and inpaints the object plus its effects.
2. A timestep-adaptive masking strategy with foreground reinitialization and attention scaling, which prevents object leakage during denoising and improves removal, especially for fast-moving objects and large objects.
3. TokSim (Token Similarity), a new object-removal metric that rewards temporal consistency among foreground tokens across consecutive frames, coherence between foreground and background tokens within a frame, and dissimilarity between the input and output foreground tokens — heavily penalizing partial or no removal.
4. WIPER-Bench, a new real-world benchmark of 60 videos covering shadows, reflections, mirrors, translucent objects, simultaneous multiple associations, and spatially disconnected associations, with results showing Object-WIPER surpasses both training-based and training-free baselines on TokSim.

Main Findings

• Highest TokSim on both benchmarks: Object-WIPER reaches TokSim 32.80 on DAVIS and 33.09 on WIPER-Bench, exceeding training-based methods such as ROSE (29.36 on DAVIS, 30.02 on WIPER-Bench), Propainter (28.24 / 20.99), VACE (15.86 / 11.53), and Gen-Prop (30.52 on DAVIS; WIPER-Bench results were not obtained — the authors report they sought results from Gen-Prop's authors and obtained them only for DAVIS). Training-free baselines scored lower (KV-Edit 23.17 / 14.46, Attentive-Eraser 30.82 / 25.28, KV-Edit-Video 28.68 / 23.26).
• Existing metrics are easy to game: Frame-wise and video VAE reconstructions that do not remove the object at all score TokSim of only 0.32 and 1.25 on DAVIS (0.42 and 0.86 on WIPER-Bench), yet their BG-PSNR, Text-align, and Quality values sit in a compressed range close to the best-performing methods.
• Best text alignment: Object-WIPER records the highest Text-align score on both benchmarks — 26.63 (DAVIS) and 26.91 (WIPER-Bench), both on a ×10² scale — indicating a high per-frame object and associated-effect removal rate.
• Strong temporal consistency on the new benchmark: Object-WIPER obtains the lowest FG-Flicker on WIPER-Bench at 3.02 (lower is better), versus 9.20 for KV-Edit and 8.92 for Attentive-Eraser; on DAVIS its FG-Flicker is 16.37.
• Background fidelity is comparatively lower: On BG-PSNR, Object-WIPER scores 23.02 on DAVIS and 27.53 on WIPER-Bench, below Propainter (34.01 / 41.07), ROSE (26.97 / 30.90), and KV-Edit (32.31 / 35.17), while the video Quality score is 61.62 on DAVIS and 61.80 on WIPER-Bench.
• Qualitative advantages over the best training-free baseline: Against Attentive-Eraser, Object-WIPER removes the object and the associated effect (reflection, translucent, mirror, shadow) in the four demonstrated cases, whereas Attentive-Eraser fails to remove the associated effect. The paper also reports that no existing method, training-free or otherwise, handles mirrored objects, and that Gen-Prop leaves remnants under fast object motion.
• Ablation results on DAVIS (TokSim / BG-PSNR / Text-align): full model 32.80 / 23.02 / 26.63; without attention scaling 32.97 / 21.92 / 26.42; without adaptive masking 32.10 / 22.73 / 26.44; without re-initialization 30.36 / 23.47 / 25.92; without the associated-effect mask 32.18 / 23.10 / 26.17. Attention biasing is credited with a 1.1 dB BG-PSNR gain, and reinitialization with eliminating residual structures.
• Metric behavior: TokSim scores high only when the object is fully removed and falls progressively as removal becomes partial; the paper contrasts this with BG-PSNR and quality scores, whose ranges remain extremely compressed across outputs of very different removal quality.

Methodology in Plain English

The method takes three inputs: a video, a binary mask marking the object to be removed in each frame, and text prompts (source and target) that describe the object and its effect — for example a "duck" and a "reflection."

Step one — finding the effect region. The video generator the authors use is a multimodal diffusion transformer where text and video tokens share one embedding space. The authors pick the text tokens that name the object and its effect, compute how strongly each visual token attends to those text tokens, and threshold that map to get a rough proposal region. Because some genuinely relevant tokens receive weaker attention, the proposal has holes; the authors fill them by checking, through visual self-attention, which remaining tokens respond most strongly to the already-identified proposal tokens. The result is the associated-effect mask, which is unioned with the user's object mask.

Step two — inversion with saved background. The source video latent is inverted to noise using an RF-Solver sampler. While inverting, the authors save the background attention values from the last several self-attention blocks and last several timesteps, and they compute a timestep-adaptive object mask, because the object's footprint in attention space grows as the latent becomes noisier. They also scale down the attention that background tokens pay to object tokens during the final inversion steps, so the background absorbs less object information.

Step three — denoising with reinitialization and attention control. The masked region is replaced with Gaussian noise, erasing all object and effect priors, while the background latent is kept. During the first denoising steps — when global structure is decided — attention from object tokens to background keys is boosted so the mask region is filled from the surrounding scene, and the saved background values are copied back outside the mask. In later steps the model denoises normally to blend everything into a coherent video.

Evaluation. The new TokSim metric uses DINOv3 token embeddings to compare output foreground tokens with neighboring frames (temporal consistency), with the input video at the same locations (removal), and with nearby background tokens (blending), combining the three into a single score scaled by 100. The authors also evaluate BG-PSNR, FG-Flicker, Text-align, and DOVER video quality. Their model is Hunyuan-T2V with 20 MMDiT blocks and 40 single blocks, 25 sampler timesteps, background values stored/copied for k = 15 timesteps across the last r = 20 single blocks, adaptive masking for 15 timesteps across all 40 single blocks, attention scaling for 10 steps with c = 0.8 and b = 1.2, CFG of 1 during inversion and 5 during denoising, and associated-mask computation at intermediate MMDiT timesteps t_i in {6, 7, 10}.

Why This Matters

This work reframes video object removal as removing an object and everything it causes — the shadow on the ground, the reflection in water, the mirror image, the light through translucent material — without paying for a training run or collecting synthetic data, which prior work such as ROSE required. It also exposes a measurement problem: metrics like BG-PSNR, quality, and even text alignment can be high while the object is still fully visible, so the paper's TokSim metric and WIPER-Bench dataset give the field a way to distinguish real removal from cosmetic edits.

Real-world applications (as framed in the paper):

• Film and video production, where boom microphones, crew, and equipment need to be removed along with their shadows.
• Surveillance and privacy protection, where people or objects should be erased without leaving traces of their presence.
• Creative content generation, where users edit footage with text-guided intent.
• Editing real-world footage containing reflections, mirrors, and translucent objects, which the paper notes no existing method — including training-based ones — handled for mirror cases.

Industry relevance: Because the approach is training-free and builds on existing pre-trained text-to-video diffusion transformers, it can be deployed on top of an already-available generative video model without the data-collection and fine-tuning costs that training-based removal methods require.

Future Directions

• Closing the background-fidelity gap. Object-WIPER leads on TokSim but trails methods such as Propainter on BG-PSNR (23.02 versus 34.01 on DAVIS), so improving background preservation without reintroducing object leakage is an open problem.
• Comparison against OmniMatte-Zero. The paper's closest competitor could not be compared because public code was unavailable; a direct head-to-head evaluation remains to be done.
• Robustness to hard cases. The paper notes failure modes of prior work under fast motion, textureless areas, and translucent objects; how Object-WIPER behaves in these regimes beyond the reported examples is not fully quantified.
• Metric and benchmark adoption. Whether TokSim generalizes to other generative video editors, and whether WIPER-Bench's 60 videos and six association categories prove sufficient, will be determined by community use — the authors state the benchmark, source code, and pre-trained models will be publicly released.

Target Audience

Researchers and practitioners working on video editing, video inpainting, and diffusion-based generative video, particularly those interested in training-free editing methods and in evaluation methodology for object removal. It is also relevant to VFX and post-production engineers who need to delete objects and their shadows or reflections from real footage, and to benchmark designers looking for an example of a curated real-world dataset with associated-effect annotations.

Authors’ abstract

In this paper, we introduce Object-WIPER, a training-free framework for removing dynamic objects and their associated visual effects from videos, and inpainting them with semantically consistent and temporally coherent content. Our approach leverages a pre-trained text-to-video diffusion transformer (DiT). Given an input video, a user-provided object mask, and query tokens describing the target object and its effects, we localize relevant visual tokens via visual-text cross-attention and visual self-attention. This produces an intermediate effect mask that we fuse with the user mask to obtain a final foreground token mask to replace. We first invert the video through the DiT to obtain structured noise, then reinitialize the masked tokens with Gaussian noise while preserving background tokens. During denoising, we copy values for the background tokens saved during inversion to maintain scene fidelity. To address the lack of suitable evaluation, we introduce a new object removal metric that rewards temporal consistency among foreground tokens across consecutive frames, coherence between foreground and background tokens within each frame, and dissimilarity between the input and output foreground tokens. Experiments on DAVIS and a newly curated real-world associated effect benchmark (WIPER-Bench) show that Object-WIPER surpasses both training-based and training-free baselines in terms of the metric, achieving clean removal and temporally stable reconstruction without any retraining. Our new benchmark, source code, and pre-trained models will be publicly available.