HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images

AI summary

Overview

Research area: Computer vision / diffusion-based image generation, specifically reference-based image inpainting applied to human-product composite imagery for advertising and e-commerce.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, DiT/MMDiT architectures, VAE token encodings, latent-space MSE training, flow matching, and standard evaluation metrics (CLIP, DINO, SSIM), though its two core technical ideas are described in largely conceptual terms.

Scope: The paper presents HiFi-Inpaint, a high-fidelity reference-based inpainting framework built on FLUX.1-Dev that inserts a product reference image into a masked region of a human image, together with a 40,000+ sample training dataset (HP-Image-40K), a Shared Enhancement Attention (SEA) module, and a Detail-Aware Loss (DAL). Accepted by CVPR 2026.

What This Paper Is About

Generating images that show a person together with a specific product is commercially important, but hard: the product's fine details (shapes, colors, patterns, textures, text, logos) must be preserved exactly, and existing approaches tend to blur, average, or hallucinate them. The authors target reference-based inpainting, where a masked human image, a product reference image, and a short text prompt are combined to produce the final composite, and they argue that current methods fall short for three reasons: a lack of diverse large-scale training data, models that do not focus on product detail preservation, and supervision that is too coarse to give precise guidance.

Key Contributions

1. HiFi-Inpaint framework. A high-fidelity reference-based inpainting framework built on a high-frequency map-guided DiT architecture derived from FLUX.1-Dev, which uses a token merging mechanism to combine the masked human image, the product image, and noisy visual tokens into joint visual tokens.

2. Shared Enhancement Attention (SEA). A module that adds a parallel branch inside each dual-stream visual DiT block to process high-frequency map tokens of the product, sharing the original branch's parameters and injecting the result back with a learnable weighting factor under an attention mask restricted to the down-sampled masked region.

3. Detail-Aware Loss (DAL). A pixel-level loss that compares high-frequency maps of the predicted and ground-truth images inside the masked region, combined with a latent-space MSE loss as the overall training objective.

4. HP-Image-40K dataset. A dataset of 40,000+ samples produced by a self-synthesis pipeline (FLUX.1-Dev diptych generation, Sobel-based segmentation, YOLOv8 plus CLIP semantic filtering, and InternVL textual filtering), with appendix statistics covering mask area ratios and product categories such as bottles, containers, jars, tubes, and dispensers.

Main Findings

• State-of-the-art on synthetic benchmark images. On a 1,000-sample held-out split of HP-Image-40K, HiFi-Inpaint attains the top CLIP-I (95.0), DINO (91.9), SSIM (63.4), and SSIM-HF (42.9), plus LAION-Aes 4.40 and Q-Align-IQ 4.36, with CLIP-T 36.1. For comparison, ACE++ scores 34.9 / 93.1 / 90.7 / 58.3 / 37.2 / 4.18 / 4.00 and Insert Anything scores 35.3 / 94.1 / 89.8 / 62.1 / 40.0 / 4.20 / 3.89.

• Strongest competitor model struggles. FLUX-Kontext records the highest CLIP-T in Table 1 (36.6) but weak visual consistency, with Table 1 listing CLIP-I 82.5 and DINO 63.1. The paper's prose separately describes FLUX-Kontext as having "notably low CLIP-I (0.712) and DINO (0.631)," figures that do not match its own table entry. Paint-by-Example is the weakest overall on visual consistency (CLIP-I 69.1, DINO 63.4).

• Qualitative advantages. HiFi-Inpaint preserves text, patterns, and branding elements and remains robust when the mask region is small, whereas FLUX-Kontext sometimes generates a standalone product instead of inpainting, ACE++ struggles with small text and intricate logos, and Insert Anything produces artifacts under small masks.

• Every component contributes. In the ablation, removing synthesized data (Scheme A: CLIP-T 35.4, CLIP-I 91.8, DINO 85.4, SSIM 57.7, SSIM-HF 38.4) versus adding it (Scheme B: 35.8 / 94.5 / 89.9 / 62.4 / 41.2) shows a clear gain. Adding DAL (Scheme C: 36.2 / 94.6 / 90.7 / 62.3 / 41.8) improves over Scheme B, and adding SEA instead (Scheme D: 35.9 / 92.2 / 87.6 / 59.8 / 40.3) also helps. The full combination (Scheme E: 36.1 / 95.0 / 91.9 / 63.4 / 42.9) is best overall.

• A learnable weighting beats a fixed one for SEA. Figure 4 reports that a learnable weighting factor produces more harmonious results, while a fixed weight causes visual artifacts and conflicts across the inpainting region.

• High-frequency extraction beats Canny edge detection. The frequency-domain high-pass filter (a circular mask of radius r applied after DFT and fftshift, followed by inverse DFT) highlights text and logos, whereas Canny detects all edges and introduces background clutter.

• Human raters prefer it. In a user study with 11 image groups and 31 valid responses, HiFi-Inpaint received 36.4% for text alignment, 41.5% for visual consistency, and 39.5% for generation quality, ahead of Insert Anything (24.9 / 21.0 / 21.6), ACE++ (20.3 / 19.6 / 22.7), and FLUX-Kontext (18.4 / 17.9 / 16.2).

• Generalization to real-world data. On a separate internal real-world test set of 2,000 samples, HiFi-Inpaint leads on CLIP-T (29.7), CLIP-I (86.8), DINO (79.8), SSIM (60.5), and SSIM-HF (44.1), while ranking third on LAION-Aes (4.27) and Q-Align-IQ (3.29) behind Paint-by-Example (4.34) and FLUX-Kontext (4.30) on aesthetics, and behind Insert Anything (3.48) on Q-Align-IQ.

Methodology in Plain English

The pipeline starts with data. Because real human-product images are expensive to collect, the authors use FLUX.1-Dev to generate diptych images with a prompt template of the form "A diptych. left: [product description] right: [product and human description]," split each diptych at its vertical boundary using a Sobel filter, and then keep only pairs whose product regions match by YOLOv8 localization plus CLIP similarity and whose extracted text matches according to InternVL. Each surviving sample yields a text prompt, a masked human image, a product image, and a target composite. An additional internal real-world set of roughly 14,000 preprocessed samples is combined with HP-Image-40K for training.

The model itself is FLUX.1-Dev. Three image-derived token sequences are concatenated: encoded tokens of the masked human image, encoded tokens of the product image, and noisy tokens of the ground-truth image. A parallel token sequence is built the same way but with the product image replaced by its high-frequency map, computed by transforming the image with a DFT, suppressing low frequencies near the center with a circular mask, and inverting back to the spatial domain. In each dual-stream visual DiT block, the high-frequency branch runs through the same parameters as the main branch, and its output is masked to the down-sampled inpainting region and added to the main branch's output scaled by a learnable factor. This is what lets high-frequency detail influence features inside the mask without disturbing the rest of the image.

Training combines two losses: the usual latent-space MSE and the Detail-Aware Loss, which compares high-frequency maps of the prediction and the ground truth inside the masked region. Optimization uses flow matching, with a learning rate of 5×10⁻⁵, a total batch size of 24, 10,000 steps, images at 1024×576, and a LoRA scaling factor α of 256 equal to the rank.

Why This Matters

The work targets a real commercial bottleneck: producing on-brand product imagery at scale while keeping the product itself pixel-faithful. Its main research contribution is showing that explicitly feeding high-frequency information into both the architecture (SEA) and the training signal (DAL) improves detail preservation over methods that rely on latent-space supervision alone.

Real-world applications:

• E-commerce listings. Automatically composing model-worn or model-held product shots from a catalog reference image and a lifestyle prompt.
• Advertising and digital marketing. Generating campaign visuals where brand text, logos, and packaging details must remain legible and accurate.
• Catalog scaling. Turning a single product photo into many scene variations across poses, lighting, and environments without reshooting.
• Cross-border or localized content. Reproducing product labeling accurately while placing it into new cultural or seasonal contexts.

Industry relevance is signaled by the author affiliations (ByteDance among them) and by the fact that the evaluation deliberately covers both synthetic and real-world internal data, which is the regime where commercial deployment would occur.

Future Directions

• Diversity and realism of outputs. The conclusion states future work will focus on enhancing the diversity and realism of generated images.
• Video generation. The authors also plan to extend the method to video, which raises open questions about temporal consistency of preserved product details.
• Reconciling quality metrics on real data. On the real-world test set, HiFi-Inpaint's LAION-Aes (4.27) and Q-Align-IQ (3.29) trail some baselines despite leading on fidelity, suggesting a fidelity-versus-aesthetics trade-off worth investigating.
• Data pipeline scalability and bias. HP-Image-40K is fully synthetic and automatically filtered; how far the filtering thresholds (CLIP similarity and textual overlap) shape the dataset's diversity, and whether that limits generalization beyond the categories listed, is not resolved here.

Target Audience

Researchers and engineers working on diffusion-based image editing, inpainting, and personalized or customized generation; practitioners building e-commerce and advertising content pipelines; and readers interested in how frequency-domain representations can be used as an architectural and loss-level signal inside DiT-based models. Familiarity with FLUX-class architectures and standard generative evaluation metrics will make the paper considerably easier to follow.

Authors’ abstract

Human-product images, which showcase the integration of humans and products, play a vital role in advertising, e-commerce, and digital marketing. The essential challenge of generating such images lies in ensuring the high-fidelity preservation of product details. Among existing paradigms, reference-based inpainting offers a targeted solution by leveraging product reference images to guide the inpainting process. However, limitations remain in three key aspects: the lack of diverse large-scale training data, the struggle of current models to focus on product detail preservation, and the inability of coarse supervision for achieving precise guidance. To address these issues, we propose HiFi-Inpaint, a novel high-fidelity reference-based inpainting framework tailored for generating human-product images. HiFi-Inpaint introduces Shared Enhancement Attention (SEA) to refine fine-grained product features and Detail-Aware Loss (DAL) to enforce precise pixel-level supervision using high-frequency maps. Additionally, we construct a new dataset, HP-Image-40K, with samples curated from self-synthesis data and processed with automatic filtering. Experimental results show that HiFi-Inpaint achieves state-of-the-art performance, delivering detail-preserving human-product images.