MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

AI summary

Overview

Research area: Computer vision — benchmark design and evaluation methodology for video subtitle removal (VSR), a video restoration task closely related to video inpainting and scene text removal. Technical level: Intermediate. The paper assumes familiarity with video inpainting architectures (flow-guided propagation, diffusion backbones such as Wan DiT and SD 1.5), full-reference image metrics (PSNR, SSIM, LPIPS, DISTS), and optical-flow-based temporal metrics (RAFT), but it introduces no new model architecture of its own. Scope (one sentence): The paper introduces MFVSR, a 3,000-video multilingual benchmark with a spatially decoupled, information-conditioned protocol for diagnosing video subtitle removal methods across restoration, preservation, temporal fidelity, and delivery coverage.

What This Paper Is About

Subtitle-removal methods are currently judged with low-resolution datasets, small paper-specific test sets, and generic video-inpainting metrics that report a single full-frame number. Those scores hide localized restoration errors, unintended edits outside the subtitle region, temporal flicker, and outright execution failures. The paper builds a large paired benchmark with controlled multilingual subtitle variation and a diagnostic evaluation protocol that separates blind from mask-guided removal and reports five spatial views plus explicit coverage and failure statistics.

Key Contributions

1. Large-scale, multi-source paired data. MFVSR contains 3,000 paired clean/corrupted videos split into three equal strata of 1,000 each: Real-HQ (OpenVidHD and Pexels, retained resolution at or above 1080p), Real-LQ (Panda-70M and Pexels, at or above 512p and below 1080p), and AI-Generated (VBench-2.0 sampled-video release and the Rapidata Sora video-generation alignment dataset). An auxiliary set of 953 naturally subtitled in-the-wild videos from Bilibili, Douyin, Xiaohongshu, Kuaishou, and YouTube adds real-world coverage but has no clean targets and is excluded from paired quantitative evaluation. Each instance pairs a subtitle-corrupted video with a clean target, a pixel-level glyph mask, frame-level BBoxes, a BBox-derived model-input mask, and rendering metadata.
2. Controlled subtitle diversity. Subtitle rendering is factorized into Spatial Layout × Temporal Behavior × Visual Style × Language Mode (2 × 2 × 3 × 2 = 24 configuration cells), with 123–126 videos per configuration. Text is drawn from FLORES-200 and fonts from Google Fonts. The benchmark covers 16 languages across five script groups and 4,505 language occurrences, including monolingual and bilingual settings.
3. Fine-grained, information-conditioned evaluation. The protocol separates blind removal (input is only the corrupted video) from mask-guided removal (input is the corrupted video plus the canonical BBox-derived mask), and ranks the two families separately. Evaluation uses five spatial views — Overall, BBox, Subtitle, BBox-NG, and Outside — grouped into Restoration Fidelity, Preservation Fidelity, and Temporal Fidelity, alongside joint coverage and full-benchmark failure statistics.
4. A diagnostic study of six representative methods. CLEAR (blind) plus ProPainter, DiffuEraser, MiniMax-Remover, OmnimatteZero, and EffectErase are run under frozen inference configurations, showing capability trade-offs and delivery gaps that full-frame metrics obscure.

Main Findings

• Decoupled views disagree with each other and with full-frame scores. Subtitle LPIPS and Outside LPIPS disagree on five of ten mask-guided method pairs; BBox TWE-Excess and BBox TC disagree on six of ten pairs. Overall LPIPS tracks Outside LPIPS more closely than Subtitle LPIPS. For example, DiffuEraser has lower Subtitle LPIPS than ProPainter but higher Outside LPIPS.
• Restoration, preservation, and temporal fidelity are non-interchangeable. On Audit300, DiffuEraser favors glyph-core reconstruction (Subtitle PSNR 21.36, LPIPS 0.269, conditioned DISTS 0.116), ProPainter favors Outside preservation and temporal consistency (Outside PSNR 34.11, Outside LPIPS 0.050, BBox TC 2.57), and MiniMax-Remover favors Outside TWE-X (0.028). EffectErase reaches favorable conditional values (Subtitle LPIPS 0.251) only on its verified prefix, over a joint coverage of 43.1%.
• The blind baseline is far behind on restoration. CLEAR scores Subtitle PSNR 11.99, Subtitle LPIPS 0.621, Outside PSNR 21.48, Outside LPIPS 0.221, BBox TC 5.00, and BBox TWE-X 15.042, with joint coverage of 85.4%.
• Higher visual difficulty does not mean worse temporal stability. Real-HQ yields higher Subtitle and Outside LPIPS than AI-generated videos for all six methods, yet lower BBox TC for five of them. Because source and resolution are coupled in Audit300, the paper treats this as a property of the joint data strata rather than an isolated resolution effect.
• Duration and style redistribute errors rather than uniformly degrading them. All five mask-guided methods show higher Subtitle LPIPS on 10–30 second clips, and Hard styles reduce their Subtitle LPIPS while increasing Outside LPIPS; CLEAR worsens on both.
• Evaluability does not imply delivery. Across all 3,000 scheduled videos, complete-output rates range from 3.9% (EffectErase) to 98.3% (ProPainter), yet every method produces evaluable output for more than 95% of instances. EffectErase's temporal coverage is 47.1%, and CLEAR's is 86.2%.
• Wider evaluable scope changes what is being measured. Extending evaluation from the common intersection to each method's native evaluable set changes both assessed scope and observed quality, in a method- and metric-dependent way; spatial and temporal quality can move in opposite directions. Native-set comparisons are not controlled cross-method rankings.
• Human judgments support the spatial dimensions more than the temporal one. Held-out human comparisons show substantially stronger human–metric agreement for Restoration and Preservation than for Temporal Fidelity, where comparisons are more often undecided. The preregistered SR-Q summary shows no consistent advantage over conventional Overall-region baselines, and is retained only as a secondary analysis.
• No single aggregate winner is established. The paper reports a capability profile rather than a single ordering, and states that gains in one capability do not imply gains in another.

Methodology in Plain English

The authors first assembled clean base videos from real and AI-generated sources and screened them automatically (decoding checks, resolution and clarity filters, OCR to flag burned-in text and URLs) and then manually, removing anything with existing subtitles, watermarks, slideshows, cartoons, or anime. Small corner logos were allowed if they could not be confused with rendered subtitles.

Each clean video then passed through a controllable rendering engine that draws subtitles from a text corpus using Google Fonts, varying four things independently: where the subtitle sits and whether it moves, how it changes while visible (static versus nine animation subtypes), how the glyphs are decorated, and whether one or two languages appear. This produces a subtitle-corrupted video plus a clean target, glyph mask, BBox annotations, and rendering metadata.

For evaluation, the protocol defines five spatial regions per frame, from the whole frame down to just the subtitle core, and three capability groups. Restoration Fidelity asks whether the subtitle-covered area was rebuilt toward the clean target. Preservation Fidelity asks whether content outside the edit box stayed unchanged. Temporal Fidelity asks whether the model introduced motion errors beyond what the target video itself contains, measured with Temporal Change Error and a RAFT-based TWE-Excess. Metrics (PSNR, SSIM, LPIPS, DISTS, RAFT) keep their released implementations; regional PSNR, SSIM, and LPIPS only change the aggregation support, and conditioned DISTS is explicitly described as a controlled-input intervention rather than an official dense DISTS variant.

Six methods were run with frozen configurations. Blind removal (CLEAR) gets only the corrupted video; the five mask-guided methods get the same canonical BBox-derived mask, adapted per method (for example, nearest-neighbor resizing with dilation 4 for ProPainter, dilation 8 for DiffuEraser, thresholding above 20 for MiniMax-Remover). Raw outputs are left immutable — missing frames are neither copied nor padded, and a metric is withheld when frame correspondence cannot be verified. Two evaluation tracks are used: Lite3000 for broad spatial and temporal measurement, and Audit300, a fixed 300-video subset selected from dataset attributes alone, without access to model outputs, metric values, rankings, or human judgments, for the more expensive structural and motion-compensated metrics. Quality numbers are video-level macro estimates with bootstrap 95% confidence intervals. Across the six baselines, retained inference logs cover 10,137.4 GPU-hours (422.4 GPU-days).

Why This Matters

Impact on research. MFVSR replaces ad hoc, paper-specific test sets of 400 videos (CLEAR, VSR-Bench-400) or short 128×128 clips (ChaLearn) with a shared 3,000-video resource that pairs clean targets with controlled multilingual subtitles, separates blind from mask-guided reporting, and accounts explicitly for partial outputs and failures. The paper's framing — that the core gap is not a lack of data but the absence of a unified protocol with information-conditioned, spatially decoupled diagnosis and explicit coverage accounting — provides a template for evaluating other localized restoration tasks.

Real-world applications:

• Restoring archive or licensed film and broadcast footage that carries burned-in subtitles from an earlier release.
• Re-purposing video across markets, where hard-coded subtitles must be removed before new captions or dubbing are added.
• Cleaning user-generated or short-video platform content (the auxiliary set is drawn from Bilibili, Douyin, Xiaohongshu, Kuaishou, and YouTube) before downstream editing or re-editing.
• Preparing clean footage for model training or dataset curation where burned-in text would otherwise contaminate learning.

Industry relevance. The benchmark's emphasis on delivery reliability — complete rates spanning 3.9% to 98.3% while every method yields evaluable output for over 95% of instances — speaks directly to production pipelines, where a method that produces partial videos is not deployable regardless of its per-frame quality. The separate blind and mask-guided leaderboard blocks also reflect practical constraints, since deployed systems may or may not have mask access.

Future Directions

1. Temporal perceptual validation remains open. Held-out human comparisons agree more strongly with the spatial components than with Temporal Fidelity, and comparisons are more frequently undecided there. The paper leaves this unresolved.
2. Consolidating a single summary score. SR-Q showed no consistent advantage over conventional Overall metrics in the preregistered comparison, so the paper keeps the three-dimensional raw profile as primary and SR-Q as a secondary summary without post-hoc refitting. A better aggregate remains an open question.
3. Scaling real-world coverage into paired supervision. The 953 in-the-wild videos are excluded from paired quantitative evaluation because they have no clean targets. Extending paired supervision to this kind of content is a natural next step.
4. Broadening the method pool and its interfaces. Only six methods are evaluated, one blind and five mask-guided, with all model interfaces frozen. As more subtitle-removal and object-removal systems appear, the same protocol can be applied — and comparisons across different native evaluable sets are explicitly noted as not controlled rankings.

Target Audience

Researchers and engineers working on video restoration, video inpainting, and scene text removal; benchmark and evaluation-methodology researchers interested in decoupled, reliability-aware protocols; applied teams in video editing, dubbing, localization, and media archiving who need to choose or audit subtitle-removal systems; and dataset curators who need to understand how controlled multilingual subtitle variation and paired supervision are constructed and validated.

Authors’ abstract

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce MFVSR, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: (1) Large-scale and diverse data, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; (2) Controlled subtitle diversity, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; (3) Fine-grained evaluation, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods.