LensErase: A Foundation Model for Erase Anything

AI summary

Overview

Research area: Computer vision / generative image editing — specifically mask-based object removal (inpainting and erasure) built on a pretrained text-to-image foundation model.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, MMDiT architectures, rotary positional encodings (RoPE), LoRA fine-tuning, and reinforcement learning for generative models.

Scope: The paper presents LensErase, a single source-image-plus-mask model intended to erase general objects, visual text, watermarks, associated effects (shadows/reflections), and multiple objects across wide ranges of scale and resolution, trained on a new 101.07M-triplet dataset, refined with SFT and RL, and released alongside a new 500-case benchmark.

What This Paper Is About

Most erasing tools are specialized: one handles small objects, another handles text, another handles watermarks, and many leave behind shadows or reflections that extend beyond the user's mask. The goal of this work is to train one general-purpose model that, given only a source image and a binary mask, reliably removes whatever is selected, removes its associated visual effects, and convincingly fills in what was hidden while leaving everything else untouched. The paper also addresses a subtler problem: because removal has many valid outcomes, judging it against a single ground-truth image is unreliable, so the authors propose both a much larger training corpus and a reference-free VLM-based evaluation.

Key Contributions

1. LensErase-100M dataset. A pre-training corpus of 101.07M source–mask–target triplets, split into General Data (77.6M), Watermark Data (9.99M), Poster Data (8.02M), Large-Mask Data (4.02M), and Visual Text Data (1.44M). The authors state this is two to three orders of magnitude larger than representative object-removal datasets, which contain roughly 100K–1M training examples.

2. Mask-Aware RoPE. A 3D rotary positional encoding that injects mask identity directly into source-image tokens by assigning each token a coordinate (m_i, h_i, w_i), where m_i ∈ {0, 1} marks masked versus unmasked tokens, with target tokens assigned a proxy modality coordinate of 2. This avoids adding a separate mask token sequence or embedding layer.

3. A training recipe combining region-weighted flow matching, SFT, and contract-guided RL. Pretraining uses a region-weighted flow-matching loss with weight λ = 5 on masked tokens; SFT runs on 140K curated examples using LoRA (80K General, 20K Watermark, 20K Visual Text, 20K Poster); RL uses 19K image–mask tasks optimized with DiffusionNFT under an "erase contract" of removal targets, associated effects, and protected objects, with a removal-aware reward gate that multiplies the base reward by 0.5 raised to the number of zero-valued Removal subcriteria.

4. TraceFusion and LensErase-Bench. TraceFusion is an inference-time post-process that estimates the effective edit region from source–output residuals and restores the original source pixels outside it. LensErase-Bench is a 500-case benchmark spanning natural images, posters, watermarks, and visual text, scored without paired ground-truth targets by a VLM judge (GPT-5.6 Sol) on Removal, Inpainting, Preservation, and Cleanliness.

Main Findings

• State of the art on all six public benchmarks. LensErase leads on OBER-Test (PSNR 34.96, FID 16.97, LPIPS 0.0256, SSIM 0.9543), RORD-Val-343 (PSNR 28.23, LPIPS 0.0601, SSIM 0.9053), RemovalBench (PSNR 28.16, LPIPS 0.1125, SSIM 0.8306), OmniPaint-Bench captured subset (PSNR 26.22, LPIPS 0.1257, SSIM 0.8521), AnimeEraseBench (PSNR 30.50, LPIPS 0.0527, SSIM 0.9398), and TextEraseBench (PSNR 36.88, FID 13.51, LPIPS 0.0339, SSIM 0.9654). The paper reports only the three lowest-mean-rank baselines per benchmark in the main table and defers full 12-baseline results to Appendix Tables 10–15.

• Large margin on LensErase-Bench. LensErase scores Removal 95.15, Inpainting 92.40, Preservation 96.35, Cleanliness 97.05, Overall 95.22, and HiFi Removal 82.80%. The strongest representative baseline shown, OmniPaint, reaches Overall 73.59 and HiFi Removal 42.80%.

• Each training stage adds measurable gains. Across the ablation, HiFi Removal rises from 65.80% after pre-training, to 70.80% after SFT, to 81.60% after RL, and to 82.80% after TraceFusion. TraceFusion also lifts mean PSNR on the public benchmarks from 28.70 to 30.82 dB and lowers LPIPS from 0.0816 to 0.0684.

• Mask-Aware RoPE beats concatenation alternatives at the pre-training stage. Against Channel Concat and Token Concat, Mask-Aware RoPE achieves Overall 86.51 / HiFi Removal 65.80 / PSNR 28.03 / LPIPS 0.0855, versus 84.32 / 58.80 / 27.84 / 0.0869 for Channel Concat and 80.78 / 51.40 / 27.41 / 0.0965 for Token Concat.

• Keeping the full source image as a visual condition is a deliberate design choice. The paper argues that masking out the selected region would discard information needed to identify associated shadows and reflections and to recover content visible through semi-transparent overlays.

• RL addresses stochastic variability rather than average quality. The authors report that the SFT model's outputs vary across samples for the same input — some clean, others leaving fragments or altering neighbors — which motivates the reward-based post-training.

• The reward gate specifically prevents compensation. A weighted average can reward an output that scores well on reconstruction or preservation despite clear removal failure; the 0.5^n₀ multiplicative penalty progressively suppresses such candidates.

Methodology in Plain English

The team started from an existing text-to-image model, Lens, and repurposed it. Because erasing is a visual-to-visual task, both the branch that normally handles text and the branch that handles images were initialized from Lens's image branch. The noisy target image goes through one branch; the intact source image goes through the other. The mask is not fed in as extra tokens but instead folded into the positional encoding, so the model can tell which source tokens are inside the selection just from where it thinks those tokens "are."

Training data was assembled rather than collected. For general objects, the pipeline inserts a scene-appropriate object into a clean photo using an editing model and then reverses the pair — the inserted image becomes the input, the original photo becomes the target — with masks derived from DCT and DINOv3 feature differences feeding SAM2. For watermarks, the watermark is rendered programmatically so the mask is known exactly. For visual text, an editing model produces a teacher edit that OCR is used to verify, followed by Poisson blending to hide seams. For posters, elements are rendered as explicit z-ordered layers and then deleted. Because prompting cannot reliably insert very large objects, a separate copy-paste pipeline with BiRefNet and YOLOE covers large masks. A further 140K high-resolution examples, screened by GPT-5.6 Sol and including professionally designed posters, were used for supervised fine-tuning.

After SFT, the model was optimized with reinforcement learning. For each task, GPT-5.6 Sol first writes a "contract" naming what must disappear, what effects must disappear with it, and what must be preserved. The model then generates several candidate outputs, each scored on removal, reconstruction, preservation, cleanliness, and an outside-fidelity PSNR term, with the removal failure penalty applied on top.

Finally, TraceFusion runs at inference. It compares the generated image against the source, finds where pixels actually changed (residuals above 4 on a 0–255 scale, keeping only components with pixels above 10), adds the dilated input mask, and checks a second time after a γ = 0.75 gamma transform to catch changes in dark regions. Everything outside that support region is replaced with the original source pixels, blended in linear RGB with a feathered boundary.

Why This Matters

Impact on research. The paper argues that single-reference evaluation is a poor fit for removal, since many plausible reconstructions exist, and offers a reference-free VLM protocol alongside the reference-based PSNR/SSIM/LPIPS/DISTS/CLIP-I/DINO/FID/CMMD metrics. It also demonstrates that a strong generative prior can be adapted to a mask-conditional editing task with a lightweight positional modification rather than architectural additions, and that reward-based post-training can be applied to removal using automatically generated contracts.

Real-world applications (as the paper frames them):

• Photography cleanup — removing unwanted objects and their shadows with a selection mask.
• Content creation and graphic design — deleting poster elements, logos, or layered design components while preserving the rest of the layout.
• Watermark and visual-text removal — recovering clean images from watermarked or text-overlaid sources, including translucent overlays where underlying content must be inferred.
• Automated visual-production pipelines — supplying a general-purpose erase step for downstream generation and editing systems.

Industry relevance. A single model covering objects, text, watermarks, posters, and associated effects replaces what would otherwise be a stack of task-specific models. The authors state the model is intended to serve as a foundational component for erase-related applications and systems.

Future Directions

• Scaling and data balance. The authors note that large-mask samples were under-represented in General Data and required a separate synthetic pipeline, and that Large-Mask Data was excluded from SFT because it is synthetic — leaving open how to obtain realistic large-object removal supervision.
• Generalizing the erase contract beyond General Data. The RL pipeline builds contracts for General Data, while Poster and Watermark rely on deterministic clean references and Visual Text on verified weak references. Extending reliable automatic contract construction and reward signals to other domains is an open question.
• Reducing reliance on a VLM judge. Both the benchmark scoring and the RL rewards depend on GPT-5.6 Sol; whether cheaper or open evaluators can substitute without degrading training or evaluation is not addressed.
• Failure modes and edge cases. The paper does not report a limitations section in the provided content, so its behavior on adversarial masks, extreme resolutions, or pathological scenes is not characterized here.

Target Audience

Researchers and engineers working on image editing, inpainting, and generative foundation models who need a mask-conditioned erasure system; practitioners building content-creation or design-automation pipelines that require removal of objects, text, watermarks, or layered elements; and benchmark or evaluation researchers interested in reference-free VLM judging for tasks with multiple valid outputs. Readers should be comfortable with diffusion/flow-matching architectures and reinforcement learning terminology.

Authors’ abstract

We introduce LensErase, a general-purpose mask-based Erase Anything foundation model designed to cover a broad spectrum of real-world erase scenarios. LensErase supports diverse objects, visual text, watermarks, associated visual effects, multiple-object removal, and a wide range of object scales, aspect ratios, and image resolutions. First, to enable such broad coverage, we construct LensErase-100M, a large-scale dataset containing 101.07 million paired source-mask-target examples spanning diverse erase scenarios. Second, we pretrain LensErase from the text-to-image model Lens, adapting its strong generative prior to mask-based erasing. To efficiently incorporate the source image and erase mask, we propose Mask-Aware RoPE, a 3D rotary positional encoding that directly injects mask information into the source-image tokens without introducing a separate mask token sequence. Finally, we refine the model through supervised fine-tuning and reinforcement learning with contract-guided, region-aware rewards, and apply TraceFusion at inference to preserve high-fidelity content outside the effective edit region. Across six public benchmarks and the proposed LensErase-Bench, LensErase achieves substantial improvements under both reference-based objective metrics and reference-free VLM evaluation. We hope LensErase can serve as a foundational model for erase-related applications and systems.