From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition

AI summary

Overview

Research area: Computer vision, specifically generative image editing — diffusion/flow-matching models for image layer decomposition (separating an image into an RGBA foreground layer and an object-removed background layer).

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiTs), latent-space tokenization, VAEs, LoRA / parameter-efficient fine-tuning, and linear-attention approximations.

Scope: The paper proposes Outpaint-and-Remove, a method that repurposes a frozen pre-trained inpainting diffusion transformer (FLUX.1-Fill-dev) for image layer decomposition via lightweight LoRA fine-tuning, a dual image-mask context design, and a multi-modal latent fusion module, trained entirely on a synthetic dataset built from public sources.

Note: the paper content provided is truncated, so some details (e.g., supplementary material referenced repeatedly for failure cases and prompt examples) are summarized only as far as reported here.

What This Paper Is About

An image can be thought of as layers — foreground objects placed over a background, sometimes occluding each other — so being able to split one photo into separate foreground and background layers enables much more flexible editing. Doing this automatically is hard because there are few dedicated methods and almost no high-quality training data. The authors observe that layer decomposition looks structurally like a combination of inpainting (filling in the background where an object was) and outpainting (extending a visible object into its occluded parts, plus a transparency channel), and ask whether an existing inpainting model can simply be adapted to do the whole job cheaply.

Key Contributions

1. A reframing of layer decomposition as in/outpainting. The paper argues that the background layer is an inpainting problem (fill the masked region) and the foreground layer is an outpainting problem with transparency constraints (recover an object's full shape, with zero alpha outside the object), which allows both outputs to be produced by one adapted inpainting backbone.

2. A data- and parameter-efficient adaptation method. Rather than fully fine-tuning a model on a large curated dataset (as LAYERDECOMP does), the authors fine-tune the input projection layer and insert LoRA layers into each attention and feed-forward layer of a frozen inpainting DiT, adding a lightweight RGBA decoding path for the foreground.

3. A dual image-mask context design plus multi-modal context fusion. Two separate image-mask context sequences (foreground c^f_{I-M} and background c^b_{I-M}) are concatenated with noisy latents, and edge, segmentation, and depth maps are summarized into a compact set of latent tokens by a Multi-Modal Latent Fusion module with approximately linear attention complexity.

4. A synthetic training corpus assembled entirely from public assets. The dataset combines imperfect real foregrounds from MULAN with over-smoothed but shape-complete generated foregrounds from LayerDiffuse, composited over OpenImages backgrounds, yielding 100,000 image–foreground–background triplets.

Main Findings

• Background removal improves over the base model and all baselines. On the 526-image MULAN test set, the method reaches PSNR 27.30, SSIM 0.93, LPIPS 0.08, and FID 25.97 — best on every metric versus SD-XL Inpainting (20.92 / 0.84 / 0.17 / 69.93), PowerPaint (23.46 / 0.76 / 0.17 / 41.67), BrushNet (21.53 / 0.85 / 0.19 / 88.46), OmniEraser (21.45 / 0.72 / 0.31 / 55.80), GeoRemover (17.19 / 0.76 / 0.32 / 92.77), Qwen-Image-Edit (19.07 / 0.64 / 0.24 / 63.49), and the unadapted FLUX.1-Fill-dev (25.59 / 0.92 / 0.09 / 35.96).

• The adaptation gain over the base inpainting model is large. Compared to FLUX.1-Fill-dev, the adapted model gains 1.71 dB in PSNR and reduces FID by 9.99, indicating the base inpainting model alone does not perform object-removal-style background completion as well.

• Foreground extraction wins on human preference, not on every image metric. On 40 collected real-world high-quality images, the method scores PSNR 28.38, SSIM 0.97, LPIPS 0.07, FID 39.52 against DiffMatte's 29.69 / 0.98 / 0.05 / 28.50 and MattingAnything's 25.11 / 0.95 / 0.09 / 56.61. In a user study with 18 independent researchers, however, the proposed method received a 59.51% preference rate versus 32.34% for DiffMatte and 8.15% for MattingAnything.

• An inpainting backbone beats a general image-to-image backbone. Replacing FLUX.1-Fill-dev with FLUX.1-Kontext-dev drops performance to PSNR 26.22 and FID 36.14, and also degrades LPIPS to 0.09 — supporting the hypothesis that layer decomposition is best treated as an inpainting adaptation.

• The foreground image-mask context prevents hallucination. Removing c^f_{I-M} costs 0.26 dB PSNR on object removal (27.04 vs 27.30), and more importantly causes the model to hallucinate or alter content inside the extracted foreground region.

• Multi-modal context helps, and each modality contributes. Removing the multi-modal context drops performance to PSNR 27.16 / FID 28.02. Ablating individual modalities gives PSNR 27.19 / FID 27.61 without edge, 27.22 / FID 28.16 without segmentation, and 27.19 / FID 28.10 without depth.

• LoRA rank must balance adaptation against prior preservation. Rank 128 is too small to learn the new task (PSNR 26.34, FID 33.92); rank 1024 overwrites pre-trained priors and causes hallucination (PSNR 27.15, FID 27.32). Rank 256 was selected as the best trade-off.

• Synthetic foregrounds in training data matter. Training without generated foregrounds yields PSNR 27.18 / FID 27.11 versus 27.30 / 25.97 with them, validating the hybrid real-plus-generated data strategy.

• The method fails on hard cases. The authors report failures on cluttered objects, large occlusions, and objects held in fingers, attributing this to the absence of such intricate samples in the synthetic training data.

Methodology in Plain English

The starting point is an off-the-shelf inpainting model that fills in masked regions of an image. The authors' key observation is that the two outputs they need already resemble what such a model can do: a background with the object erased looks like an inpainted image, and a foreground object extended into its occluded or hidden parts looks like outpainting, with an extra alpha channel marking which pixels are actually part of the object.

Concretely, they freeze the pre-trained inpainting transformer and make three additions. First, they feed the model more context: alongside the usual background image-and-mask input, they add a parallel foreground image-and-mask input, and they also compute edge, segmentation, and depth maps from the input image. To keep this from blowing up compute, they pass all those extra image-like maps through a small fusion module that summarizes them into a compact set of latent tokens using a small number of learned query tokens — attention cost grows roughly linearly with the number of input tokens instead of quadratically. Second, they fine-tune cheaply: they train the input projection layer and insert small LoRA adapters into every attention and feed-forward layer, leaving the base weights untouched. Third, they decode two outputs: the background uses the original RGB VAE, while the RGBA foreground uses a separately fine-tuned encoder/decoder.

Because no clean large-scale layered dataset exists, they build their own training triplets from public resources. They take real foreground cutouts from MULAN (detailed textures but often incomplete shapes) and generated foregrounds from LayerDiffuse guided by ChatGPT-4o prompts (complete shapes but over-smoothed textures), place them over OpenImages backgrounds at rescaled sizes with 1–3 objects per image and deliberate occlusions, and use imperfect masks on purpose so the model learns to infer accurate object boundaries. Evaluation uses PSNR, SSIM, LPIPS, and FID on MULAN's 526 test images plus a user study on 40 real-world images.

Why This Matters

Impact on research. The paper's argument is that a whole new generative capability does not require training a new model from scratch — a pre-trained inpainting model plus a light adaptation, a fusion module, and better context conditioning can be sufficient. That is a meaningful reframing of how a multi-output task can be decomposed into variants of a single existing task, and it makes layer decomposition reproducible for groups without industrial-scale compute or proprietary data. It also positions the approach as "plug-and-play": as better inpainting models appear, the same adaptation recipe could be reapplied.

Real-world applications:

• Photo and design editing: isolating a subject so it can be moved, resized, or recomposed over a new background without hand-drawn masks or manual cleanup.
• E-commerce and product imagery: generating clean product cutouts with transparency and consistent backgrounds from ordinary photographs.
• Content creation pipelines: layered artwork generation and asset extraction where foreground and background need to be edited independently.
• Component-based retrieval: indexing and searching for objects as separate layers rather than as pixels inside a flattened image.

Industry relevance. The commercial value lies in flexibility: the training corpus is built entirely from public datasets and tools, and the adaptation is parameter-efficient, so the approach lowers the barrier for teams that lack the resources to fully fine-tune a large generative model. That said, the paper's own foreground metrics show the method does not dominate a specialized matting baseline like DiffMatte on PSNR, FID, or LPIPS — its advantage there is human-perceived quality, which matters for creative tools but is a weaker guarantee for automated pipelines.

Future Directions

1. Improving the training data for hard cases. The authors explicitly attribute failures on cluttered scenes, large occlusions, and objects held in fingers to missing intricate samples, and expect better data to alleviate this.

2. Extending evaluation beyond MULAN. MULAN is described as the only standard dataset in this domain, and its layers are generated by off-the-shelf detection, segmentation, or inpainting models with unguaranteed quality — so better benchmarks and ground-truth layer annotations remain an open need.

3. Revisiting the LoRA rank trade-off at larger scale. Since rank 1024 overwrote pre-trained priors while rank 128 underfit, it is unclear how the optimal rank shifts if the adaptation data becomes more commercial-grade.

4. Testing whether the recipe transfers to other inpainting backbones. The paper demonstrates the approach on FLUX.1-Fill-dev and compares against FLUX.1-Kontext-dev; whether the same fusion and dual-context design works as well on other inpainting models is not established.

Target Audience

Researchers and practitioners working on diffusion-based image editing, object removal, matting, and layered/controllable generation — particularly those interested in parameter-efficient adaptation of large pre-trained models. It is also relevant to applied engineers building editing or asset-generation pipelines who need strong results without large-scale training budgets. Readers without a background in diffusion transformers and LoRA fine-tuning will find the technical sections demanding.

Authors’ abstract

Images can be viewed as layered compositions, foreground objects over background, with potential occlusions. This layered representation enables independent editing of elements, offering greater flexibility for content creation. Despite the progress in large generative models, decomposing a single image into layers remains challenging due to limited methods and data. We observe a strong connection between layer decomposition and in/outpainting tasks, and propose adapting a diffusion-based inpainting model for layer decomposition using lightweight finetuning. To further preserve detail in the latent space, we introduce a novel multi-modal context fusion module with linear attention complexity. Our model is trained purely on a synthetic dataset constructed from open-source assets and achieves superior performance in object removal and occlusion recovery, unlocking new possibilities in downstream editing and creative applications.