Editing Images into Reusable Layers via Compositional Edit-Prior Distillation

AI summary

Overview

Research area: Computer vision, specifically instruction-driven image editing and layered (RGBA) image generation.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching transformers, LoRA adaptation, latent-space tokenization, alpha compositing, and on-policy distillation.

Scope: The paper introduces LayerEditor, a unified model that takes a source image, a text editing instruction, and a requested layer count, and produces both an edited canvas and an ordered stack of reusable RGBA layers whose recomposition should match the edit.

What This Paper Is About

Most instruction-based image editors output a single flattened image in which text, objects, and background are fused, so any later revision means regenerating the whole image and risking damage to untouched regions. Existing decomposition models can recover layers, but they are trained separately from editing, so a visually plausible edit can still yield messy or unusable layers. The paper's goal is to do both jobs at once: apply the requested edit and emit a clean, ordered set of reusable RGBA layers, while keeping the editing quality of the pretrained model it is built on.

Key Contributions

1. LayerEditor, a unified layered editing framework. A single model that jointly generates an edited canvas and an ordered sequence of RGB–Alpha layers, supporting variable layer counts and user-specified content for individual layers.

2. In-Context RGB–Alpha Token Modeling. The edited canvas and all layer appearance/opacity fields are placed in one token sequence with shared attention and layer-aware rotary coordinates, where source-image tokens use slot s = −1, the canvas uses s = 0, and layer i uses s = 2i+1 (RGB) and s = 2i+2 (Alpha). A dedicated Alpha decoder, initialized from the pretrained RGB decoder, reconstructs continuous opacity.

3. Compositional Edit-Prior Distillation (C-EPD). An on-policy distillation scheme that queries a frozen copy of the pretrained editor at intermediate states of the student's own denoising trajectory and supervises both the direct edited canvas and the image composed from the predicted layers, so the pretrained editing prior survives adaptation to layered outputs.

4. LayerEditor-100K plus two benchmarks. A 100K paired dataset with ordered RGBA annotations, built from Crello, PrismLayersPro, and ScaleEdit, together with Crello-Testset (2,994 cases) and ScaleEdit-Testset (1,000 cases) for evaluating layered editing, layer-count control, layer-content control, and image-to-layer decomposition.

Main Findings

• Strongest recomposition fidelity on both benchmarks. LayerEditor reaches Composite SSIM 0.8351 and Composite FID 33.86 on Crello-Testset, and Composite SSIM 0.8839 / Composite FID 39.48 on ScaleEdit-Testset, the best Composite SSIM/FID in the comparison table on both.

• Gains over the strongest API baseline. Against Seedream5-Pro (Joint), LayerEditor improves Crello Composite SSIM by 5.7% and reduces Composite FID by 38.8%. On ScaleEdit-Testset, Composite SSIM rises from 0.8365 to 0.8839 and Alpha IoU from 0.6583 to 0.6802.

• Smaller quality drop from canvas to layers. On ScaleEdit-Testset, Seedream5-Pro (Seq.) falls from 0.8933 Direct SSIM to 0.7901 Composite SSIM, an 11.6% reduction, whereas LayerEditor falls from 0.8925 to 0.8839, a 1.0% relative reduction.

• Better layer quality by rubric. LayerEditor averages 6.00 across the five rubric dimensions (edit consistency 6.14, visual quality 6.35, layer segmentation 6.83, layer composition 5.64, Alpha accuracy 5.05) on Crello-Testset, versus 4.50 for Seedream5-Pro (Joint), the highest baseline mean. Its layer-segmentation score exceeds Seedream5-Pro (Joint) by 1.61 points and its layer-composition score exceeds FR + QIL-Decomp-FT by 1.29 points.

• Layer-count controllability. Under the controlled regrouping setting built from the pooled 3,994 cases, LayerEditor achieves Alpha IoU 0.6196 and RGBA L1 0.0460, exceeding Seedream5-Pro (Joint) by 0.0493 in Alpha IoU and 0.97 points in semantic grouping, with rubric scores of 6.88 (semantic grouping), 6.28 (composition), and 6.98 (Alpha).

• Large latency advantage. Observed latency is 41.5 s/sample for LayerEditor versus 250.0 s for Seedream5-Pro (Joint), 322.6 s for Seedream5-Pro (Seq.), and 104.2 s for FR + QIL; the paper notes this was measured without controlled hardware.

• Transfers to pure decomposition without I2L fine-tuning. On 1,171 held-out Crello images with 2–10 reference layers, LayerEditor gets Composite SSIM 0.8761 versus 0.7963 for QIL, with Alpha IoU 0.6343 and layer-semantic score 6.39.

• Each component is load-bearing (ablation on Crello-Testset). Removing C-EPD drops edit consistency from 6.14 to 4.67 and layer segmentation from 6.83 to 5.01. Removing only the Direct or only the compositional guidance term improves edit consistency relative to no C-EPD but lowers visual quality and layer segmentation and raises RGBA L1; combining both improves all four metrics. The dedicated Alpha decoder cuts RGBA L1 from 0.0578 to 0.0350 and raises layer segmentation from 5.67 to 6.83 compared with averaging the pretrained RGB decoder's three channels.

• Decomposition fine-tuning alone helps but is not enough. Fine-tuning a decomposition model on the new data raises Composite SSIM from 0.5584 to 0.6343 on Crello-Testset and from 0.6451 to 0.6927 on ScaleEdit-Testset, still below LayerEditor.

• Dataset composition. LayerEditor-100K contains 48,329 text edits, 36,858 object edits, and 14,813 poster-level appearance edits, with 2–10 layers per target.

Methodology in Plain English

The team starts from an existing strong instruction-based editor, FireRed-Image-Edit-1.1, and freezes its encoders and RGB decoder, adapting only the backbone with rank-256 LoRA. Instead of asking the model to output one image, they restructure the output as one long latent sequence: first an "edited canvas" slot, then alternating RGB and Alpha slots for every layer. A slot index rides along with each token so the model knows which field a token belongs to and how layers stack, while the usual x–y coordinates keep spatial alignment. Because every field is denoised together in one sequence with shared attention, the canvas and all layers stay consistent with each other, and the number of layers can vary without changing the architecture.

Opacity gets special treatment. A separate Alpha decoder, copied from the pretrained RGB decoder but with a single-channel head, is trained on its own with reconstruction, composition, perceptual, and adversarial terms across three stages (single-layer reconstruction, full-document supervision, latent-noise adaptation). It is then frozen. Using a dedicated opacity path avoids approximating alpha by averaging the RGB decoder's three channels, which the ablation shows is noticeably worse.

The trickiest problem is that fine-tuning an editor to emit layers damages the editing skill it started with. The fix, C-EPD, keeps a frozen copy of the original editor as a teacher and, during training, lets the student run partway down its own denoising trajectory. At sampled intermediate states the teacher is asked what the edited image velocity should be, giving supervision at states the student actually visits rather than at states a perfect teacher would visit. A second loss decodes the student's partially denoised layers, composites them back-to-front using the standard alpha-over formula starting from a white background, and matches that composite to the teacher's decoded prediction. Both losses are added to ordinary flow matching, with the C-EPD weight ramping up as 0.1·min(k/500, 1) and composite weight 0.05. At inference the teacher is gone: one 30-step trajectory with CFG 4.0 produces everything.

For data, the authors reuse existing layered design sources where layer structure already exists and, for flattened editing pairs, recover target layers through planning, decomposition, refinement, and quality filtering, verifying that instructions are executed, layers are isolated, and recomposition is faithful.

Why This Matters

The paper argues that flat outputs are a bottleneck for any workflow where an edit is a starting point rather than an endpoint. By making layers a first-class output of editing, revisions can be localized to one element instead of regenerating the whole composition.

Real-world applications:

• E-commerce content production, where a product, price text, and background badge need independent revision.
• Advertising and poster design, where individual text rows, objects, and decorations get reorganized across variants.
• Template-driven design tools that need editable, composable assets rather than a rendered picture.
• Asset reuse pipelines, where an element edited once is carried into new layouts.

Industry relevance: The reported latency of 41.5 s/sample versus 250–323 s for the strongest API baseline matters for production loops, and the ability to control both layer count and layer content maps directly onto how design tools already structure documents. The result that a single model can also do plain image-to-layer decomposition without task-specific fine-tuning reduces the number of separate models a product team has to maintain.

Future Directions

• Scaling beyond 10 layers. The dataset and evaluation cover 2–10 layers; whether the single-sequence formulation stays coherent for much deeper stacks is untested here.
• Extending beyond static design imagery. Training data is drawn from design and poster sources; performance on natural photographs, video frames, or 3D-consistent content is not reported.
• Reducing the reliance on recovered supervision. Flattened ScaleEdit pairs acquire target layers through planning, decomposition, refinement, and filtering, so the ceiling of the approach is partly tied to the quality of that recovery pipeline.
• Understanding where distillation substitutes for data. C-EPD recovers editing fidelity from a frozen teacher; how much of the layered behavior could instead come from more or better paired layer annotations is left open.

Target Audience

Researchers and graduate students working on controllable image generation, layered/transparent image representations, and distillation for diffusion or flow-matching models. It is also relevant to applied engineers building design, e-commerce, or advertising tools who need editable rather than flattened generative output, and who care about preserving a pretrained editor's behavior while changing its output format.

Authors’ abstract

High-quality image editing requires precise modification of target regions while preserving the content of unrelated regions. Existing editing models produce flattened outputs that limit precise local manipulation and layer reuse, while decomposition methods typically optimize layer recovery separately from instruction-driven editing. To this end, we introduce LayerEditor, a unified framework that couples instruction-guided image editing with controllable generation of reusable RGBA layers. At its core, In-Context RGB-Alpha Token Modeling jointly represents the edited canvas and an ordered sequence of RGB-Alpha layers within a single token sequence, enabling semantic edits and explicit control over individual layer content. To retain the pretrained model's editing capability during adaptation to layered outputs, we propose Compositional Edit-Prior Distillation, which transfers its editing prior to both the edited canvas and the image composed from the predicted layers. To support training and evaluation, we construct LayerEditor-100K, a large-scale paired image-editing dataset with ordered RGBA layer annotations, and introduce dedicated benchmarks for layered editing. Extensive experiments demonstrate that LayerEditor maintains strong editing fidelity while generating accurate RGBA layers, supporting both flexible layer-level manipulation and multi-layer decomposition.