DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal

AI summary

Overview

Research area: Computer vision — video restoration and generative editing, specifically removing text overlays (subtitles, titles, watermarks, logos) from video while reconstructing the scene underneath.

Technical level: Advanced. The paper assumes familiarity with latent video diffusion/flow models, VAEs, LoRA fine-tuning, and classifier-free guidance, though the core ideas are describable in plain terms.

Scope: The paper introduces DVTE-Bench, a paired benchmark of 1,631 videos across eight text-overlay categories, and DeOverlay, a source-anchored flow model that reconstructs clean video starting from the source latent with fixed source guidance.

What This Paper Is About

Videos often carry subtitles, titles, watermarks, and logos that cover parts of the scene. Removing them requires more than blurring pixels: the covered content must be reconstructed so it matches what was actually there, while everything around it stays unchanged and temporally stable. The paper argues that two existing families of methods fall short — propagation-based inpainting (which has no direct observation when a region stays covered) and noise-initialized generative editors (which discard the source structure at the start and can leave residual lettering or distorted boundaries). The authors respond with a benchmark for comparing methods under varied overlay conditions and a model that begins from the source latent and keeps the original video available at every update step.

Key Contributions

1. DVTE-Bench, a paired benchmark of 1,631 clips (136,090 frames, 91.1 minutes; 1,085 portrait and 546 landscape) spanning eight text-overlay categories, with OCR-screened clean targets, frame-aligned pixel masks (including masks that follow moving text), 11 language labels, and a shared evaluation protocol. Table 1 contrasts it with CLEAR's 400-clip test set and VSR-Bench-400 (400 clips), neither of which is marked as released in the comparison.

2. DeOverlay, a source-anchored flow model. Training connects source and clean latents with a paired flow path and a region-weighted flow-matching objective (λ = 5, so covered elements receive weight six and exterior elements weight one); inference starts directly from the source latent and applies successive shared-weight Euler updates.

3. Persistent source guidance without a second token stream: visibility-aware source aggregation (VSA), global source conditioning (GSC), and a mask embedding are added to the current state tokens, preserving a single video-token sequence. The paper measures that a source-token stream would raise tokens from 13,346 to 26,594, median time by 2.485×, and incremental peak memory by 1.995×.

4. Full-release evaluation plus ablations: DeOverlay beats four baselines on all five reconstruction and perceptual metrics, and component and step-count studies isolate the effect of source initialization, each guidance branch, and the number of flow updates.

Main Findings

• Best on all five metrics across all 1,631 clips. DeOverlay (K = 6) reaches Mask PSNR 30.80, Mask SSIM 0.8822, Mask LPIPS 0.0408, DISTS 0.0507, and VFID 0.0835. The strongest baseline on these metrics, MiniMax-Remover, records 26.17, 0.8044, 0.0977, 0.0856, and 0.2806; ProPainter records 25.90, 0.7832, 0.1166, 0.0957, 0.3879; DiffuEraser 24.84, 0.7723, 0.1130, 0.0952, 0.3285; and CLEAR 16.02, 0.5172, 0.3209, 0.2158, 1.4736.

• Large relative reductions versus MiniMax-Remover: Mask LPIPS down 58.3%, DISTS down 40.8%, VFID down 70.3%, while Mask PSNR rises from 26.17 to 30.80 dB and SSIM from 0.8044 to 0.8822.

• Temporal diagnostics. DeOverlay obtains TWE Pred 9.3110 against a clean-target reference of 9.3228, giving an aggregate TWE Gap of 0.0118, versus 0.6133 for MiniMax-Remover and 1.1801 for ProPainter. ProPainter's TWE Pred (8.1428) and TC (4.0349) are lower than DeOverlay's (9.3110 and 4.2498), and the paper explicitly notes that scoring below the clean-target reference does not by itself indicate better reconstruction, so the five target-referenced metrics remain the primary comparison.

• Difficulty depends on the metric. DeOverlay leads on PSNR, SSIM, LPIPS, and DISTS across all eight overlay types. Artistic titles give its lowest PSNR/SSIM (26.24 dB / 0.8409), while vertical text gives its highest LPIPS/DISTS (0.0975 / 0.1015).

• Source initialization plus additive guidance is the biggest design lever. Swapping them for noise initialization with concatenated source tokens costs 2.861 dB in Mask PSNR (29.600 versus 32.461). Within the source-initialized model, keeping only the mask embedding costs 0.231 dB; removing GSC costs 0.171 dB; removing VSA costs 0.091 dB; replacing visibility-normalized aggregation with an ordinary temporal mean costs 0.101 dB and raises MAE to 4.360.

• Repeated updates help; returns diminish. Four-step flow exceeds a separately trained direct-residual reference by 1.161 dB in Mask PSNR (32.461 versus 31.300) and improves Crop SSIM (0.92765 versus 0.91120). Within a fixed checkpoint, going from two to four steps improves Mask PSNR from 32.410 to 32.461; eight steps add only 0.007 dB over four at 1.50× latency.

• Speed and memory. On eight shared clips (650 frames) on one A100, one-step DeOverlay reaches 3.0847 FPS (1.17× MiniMax-Remover's throughput) and four steps reach 2.0712 FPS, both at a sampled peak of 12.236 GiB, compared with the lowest baseline value of 10.776 GiB for MiniMax-Remover.

• Qualitative behavior. Example comparisons show residual text, blurred structures, and altered colors in baseline outputs, while DeOverlay keeps restored structures (for example a car and road in an artistic-title case) aligned with the clean target; twelve application examples cover subtitle languages and density, title layouts, watermarks, a sticker, and a television logo.

Methodology in Plain English

The method reframes removal as transporting a video from an "overlaid" state to a "clean" state rather than generating from scratch.

1. Start where the answer already is. Both the overlaid clip and its clean counterpart are compressed into latent representations by a video VAE. Training draws a straight path between the clean latent and a lightly perturbed version of the source latent (perturbation σ = 0.05); the network learns to predict the direction of travel along that path, sampled at intermediate times. At inference no noise is used at all — the process begins at the source latent itself.

2. Keep the source in view at every step. Each update adds three fixed signals to the current state tokens: a local cue from nearby frames, a global summary of the whole clip, and a learned marker for the edit region. Because these are added rather than concatenated, the model keeps the same number of video tokens as a plain video transformer.

3. Make the local cue mask-aware. Naive temporal averaging mixes overlay pixels into useful scene information. Visibility-aware aggregation looks at a window of five latent positions (t−2 to t+2), throws out masked positions, and divides by the count of visible observations (with a 1e-4 floor) so masked spots do not shrink the average. The result is projected through two layers and gated to a dilated edit region.

4. Emphasize the region that matters. The flow-matching loss multiplies covered elements by six (λ = 5) relative to exterior elements, using the annotated mask dilated by one latent cell, so the network focuses on correction rather than copying.

5. Multiple corrections from one checkpoint. Inference applies K Euler steps with a linear–quadratic schedule, keeping weights, source, and mask fixed while state and time change. Classifier-free guidance at scale g = 2 means two network evaluations per step, so six steps require 12 evaluations. The backbone is a frozen dense 1.3B LingBot-Video model adapted with rank-256 LoRA, with the patch embedding, guidance modules, and mask embedding trained directly.

6. Build a benchmark that separates the hard cases. The eight overlay categories are organized by spatial layout (vertical text, non-bottom placement, multiple regions), temporal behavior (moving text, persistent watermarks), and appearance (standard subtitles, artistic titles with outlines or glow, speech-aligned captions). Since a clean reference containing its own text could be confused with an unerased overlay, every target frame is OCR-screened, rejecting 569 of 2,200 candidates and keeping 1,631 (74.1%).

Why This Matters

Impact on research. Overlay removal has been evaluated in fragments — subtitle erasure and visible watermark removal as separate tasks with separate data. DVTE-Bench unifies eight overlay conditions under one paired protocol with pixel-accurate scoring masks supplied independently of the bounding boxes given to mask-conditioned models, which makes cross-method comparison possible. The paper's design choice of starting from the source latent, rather than from noise, also offers a concrete alternative to noise-initialized editing for restoration tasks, and the ablations quantify what that choice is worth.

Real-world applications:

• Restoring archival and broadcast footage where burned-in subtitles, station logos, or time-stamps obscure the picture.
• Re-purposing licensed video across markets, where localized subtitles or legacy branding must be removed rather than covered.
• Cleaning user-generated and stock footage whose watermarks or stickers block the scene.
• Preparing footage for downstream vision tasks (detection, tracking, 3-D reconstruction) that are confused by overlaid text.

Industry relevance. The efficiency measurements are reported in deployment terms: 3.0847 FPS at one step and 2.0712 FPS at four on a single A100 for 650 frames, versus 2.3413 FPS for ProPainter and 2.6405 FPS for MiniMax-Remover, plus peak memory of 12.236 GiB against ProPainter's 34.101 GiB. The paper also states that deployment derives text masks from OCR and that non-text overlays require a mask covering the designated graphic region, which describes a practical pipeline rather than only a research setting.

Future Directions

• Graphic overlays and natural scene text. The benchmark is explicitly limited to synthetic text overlays; the authors list graphic logos, stickers, and preservation of natural scene text as needing broader paired evaluation, even though qualitative examples show logo and sticker removal.

• Fully occluded regions. When no neighboring frame reveals the covered background, reconstruction falls back on the video prior. Reducing dependence on that prior — or quantifying when it fails — remains open.

• Why more steps help, and how much. The appendix shows an Euler error bound under assumed Lipschitz and smoothness constants that were not measured, and the step sweep shows gains flattening (eight steps add 0.007 dB over four). Measuring those constants, or designing schedules that reach the same quality in fewer evaluations, is a direct next question.

• Residual artifacts and unexecuted comparisons. The authors report residual graphics and local texture artifacts in application examples, and note that SEDiT could only be compared conceptually because an executable model was unavailable for matched evaluation — a gap a future controlled comparison could close.

Target Audience

Researchers and engineers working on video restoration, inpainting, and generative video editing who need either a benchmark for overlay removal or a worked example of source-initialized flow models with additive conditioning. The paper is also useful to practitioners deciding between propagation-based, subtitle-specific, and generative pipelines, since it reports regional accuracy, perceptual quality, temporal diagnostics, speed, and memory on a shared protocol. Readers without background in flow matching or latent video models will find the benchmark and results sections accessible, but the method sections require that background.

Authors’ abstract

Video overlay removal aims to recover scenes obscured by subtitles, titles, and logos while preserving surrounding content. Existing methods use temporal propagation or source-conditioned generation, but can leave residual overlays and distort scene structure when useful source information is obscured. We propose DeOverlay, a source-anchored flow model that reconstructs the clean video from the source latent with fixed source guidance. Specifically, we train a shared velocity network along paired source-to-clean paths using region-weighted flow matching, enabling successive cor- rections from intermediate states. To select local source cues, we develop visibility-normalized temporal aggregation that excludes masked positions and averages the remaining observations. Global source context and edit-region embeddings complement these cues through additive condi- tioning, retaining a single video-token sequence. To evaluate scene recovery across diverse overlay conditions, we introduce DVTE-Bench, comprising 1,631 paired videos across eight text-overlay categories, with OCR-screened clean targets, frame-aligned pixel masks, and a common evaluation protocol. Experiments on the complete release show that DeOverlay outperforms four baselines on all five reconstruction and perceptual metrics, reducing Mask LPIPS and Mask VFID by 58.3% and 70.3%, respectively, relative to the strongest evaluated baseline. Additional qualitative examples demonstrate the removal of graphic logos and stickers. We will make the complete benchmark, evaluation code, and test outputs availabl. Project page:https://yahooo-m.github.io/DVTE-Bench/