Instance-Level Generation for Representation Learning

AI summary

Overview

Research area: Computer vision, specifically instance-level recognition (ILR) and image retrieval, combined with synthetic training data generation via large language models (LLMs) and generative diffusion models (GDMs).

Technical level: Intermediate. The paper assumes familiarity with retrieval metrics such as mAP and recall@k, metric learning losses, and vision-language foundation encoders such as SigLIP and CLIP, but the pipeline itself is described in accessible stages.

One-sentence scope: The paper proposes a pipeline that takes only the name of a target domain as input and generates a fully synthetic, instance-labeled training set used to fine-tune a single vision encoder for instance-level retrieval across seven benchmarks.

What This Paper Is About

Instance-level recognition identifies individual objects rather than broad categories, which makes it the most granular form of image classification and, consequently, extremely expensive to annotate at scale. Because ILR requires exhaustive instance-specific labels and images of the same object under different conditions, large-scale training data has remained a major bottleneck, and multi-domain datasets often lack dedicated training sets altogether. This paper's goal is to remove the need for real images entirely by automatically generating object instances under diverse conditions and backgrounds, then fine-tuning a foundation vision model on that generated data.

Key Contributions

1. A four-stage instance-level generation pipeline requiring no real images. Given only a domain name or textual description, an LLM produces a list of object categories, a GDM generates object instances, backgrounds and lighting are synthesized per instance, and geometric augmentations are applied. The authors state this is the first method tailored specifically for ILR synthesis.

2. A new ILR paradigm with minimal input. The only input is the name of one or more target domains (for example "everyday objects" or "artworks"), and the output is a representation model fine-tuned for those domains, as opposed to methods that fine-tune per instance using a few real images.

3. A single model that generalizes across diverse ILR domains. Rather than one model per domain, the approach learns one representation that improves over the pretrained backbone across artwork, landmark, product, and multi-domain benchmarks.

4. Synthetic data positioned as both a replacement for and a complement to real data. The paper provides a comparison against real manually labeled training sets and a mixture experiment showing that combining real and synthetic data outperforms using either alone.

Main Findings

• Consistent gains over the pretrained backbone: Fine-tuning SigLIP on ILGen-ALL (ID7) improves over the pretrained model (ID1) on all seven evaluation columns, with an average improvement of 5.2 (from 47.5 to 52.7). Per-dataset: MET 67.3 to 75.1, ROP 45.0 to 48.6, GLDv2 15.7 to 18.7, SOP 55.4 to 55.6, INSTRE 80.6 to 87.5, mini-ILIAS 21.0 to 30.6.

• Synthetic generation beats rendered 3D objects on most benchmarks: ILGen-ALL achieves a 52.7 average versus 50.9 ± 0.5 for Objaverse-background, which renders 20,000 of the 818K objects in Objaverse 1.0 into views with the same background process, with the largest gap on SOP. The authors conclude the method learns representations at least as effective as those learned on rendered objects.

• Synthetic beats real labeled data in overall average: ILGen-ALL reaches 52.7 average versus 51.4 for Real-ALL, which merges training sets from SOP, InShop, RP2k, GLDv2, and MET at 20K instances with 4 images per class. Per-domain, training on real single-domain data wins within its own domain, but the synthetic model performs better across other domains except product. Performance on ROP is always better when training on real images, which the authors attribute to possible shortcomings of generative models for large objects with many details.

• Real plus synthetic is the best configuration: Mixing progressively more real data on top of 20,000 synthetic instances raises the average from 51.4 (real only) and 52.7 (synthetic only) to 53.5 at 2,000 real instances, 53.9 at 4,000, 54.2 at 8,000, 55.3 at 12,000, 54.0 at 16,000, and 55.0 at 20,000. Balanced or nearly balanced mixing performs best overall, and products are the one domain where mixing is slightly worse than real-only.

• Instance count saturates: With ILGen-G, 5K generic instances already give 51.2 average, 10K gives 51.6, and 20K gives 50.8, so average performance increases from 5K to 10K then saturates. With 5K, every benchmark improves over pretrained except SOP where performance does not change.

• Diverse backgrounds matter, with one exception: ILGen-ALL without background generation (ID6) gives 49.5 average and drops on INSTRE (77.3) with only a small gain on mini-ILIAS (21.5). Adding synthesized backgrounds (ID7) substantially improves most datasets, but SOP is the exception: the clean-background variant reaches 61.1 versus 55.6 with diverse backgrounds, which the authors relate to the commonly clean backgrounds in that test set.

• Domain-specific instances beat generic ones at equal size: Complementing ILGen-G-10K with 10K domain-specific objects (ID7) is better on average than complementing it with 10K generic objects (ID5), strengthening all test sets except SOP.

• Gains hold per query: On a query-by-query comparison for mini-ILIAS, INSTRE, and MET, fine-tuning on ILGen-ALL improves the majority of queries across the whole range of pretrained performance, including many highly performing INSTRE queries.

• Transfer across backbones: Fine-tuning ILGen-ALL improves SigLIP (47.5 to 52.7), CLIP (37.5 to 46.8), and ViT-B (25.7 to 34.3) on average. The stated exception is ViT on SOP, where performance drops from 43.7 to 40.8.

• Loss function robustness: The recall@k loss gives the best average (52.7), followed by infoNCE (52.2), softmax margin (51.5), and contrastive (50.6), showing the generated set works with a diverse set of losses.

• Ablation highlights: Using 3 positive images per class gives 51.5 average versus 50.3 with 2. A template prompt yields 52.6 and GPT-4o, DeepSeek, and Claude all land between 52.5 and 52.6. Stable Diffusion v2.0 with 50 inference steps gives 51.8, while Stable Diffusion Turbo with 5 steps gives 53.0. Substituting Stable Diffusion v2.0 as the background model causes a large drop to 47.1, and disabling padding gives 51.5.

Methodology in Plain English

The pipeline turns a domain name into training data in four stages. First, an LLM (GPT-4o) is prompted to list object category names; for the generic case the prompt is "Provide a raw list of names of everyday objects," and for specific domains the prompt is enriched with examples. Second, Stable Diffusion Turbo generates K images per category using different random seeds, with "in a clean background" appended to the prompt; the authors assume different random seeds define distinct instances, treating each of the C × K generated images as its own class. Third, background removal using RMBG v1.4 (based on IS-Net) produces a soft alpha matte, random padding of up to half the image width or height changes object size and position while preserving aspect ratio, and ICLight relights the object and synthesizes a new semantically appropriate background; this last step is repeated N = 4 times per object, and those four images form the positive set for that class. Fourth, simple random geometric augmentations are applied during training to vary viewpoint.

The generated dataset in the default ILGen-ALL configuration contains 20,000 objects: 10,000 from the generic domain (2,000 categories × 10 instances) plus 3,000 artwork (200 × 15), 4,000 landmark (50 × 80), and 3,000 product (200 × 15) instances. Training uses a query-versus-database scheme: each batch samples B = 400 classes and all their N = 4 images, giving 1,600 images per batch, with one image per class randomly chosen as the query and the rest forming the database. The loss is a differentiable approximation of recall at the top-k ranks (with k = {1, 2, 4, 8} and temperatures 0.01 and 1.0), a metric learning approach adopted from Patel et al. (2022). Training uses a learning rate of 10⁻⁵, the Adam optimizer with weight decay 10⁻⁶, random cropping, resizing, flipping, color jitter, and grayscale augmentation, and runs on a single A100 or V100 GPU; ILGen-ALL with SigLIP takes approximately 2.5 hours on an A100.

Why This Matters

The work reframes a data-scarcity problem as a generation problem: instead of collecting and annotating instance-level images, the only required input is a domain name. It establishes that synthetic generation, without any real images, can serve as a practical complement to real training sets for retrieval across artwork, landmark, product, and multi-domain benchmarks.

Impact on research:

• Provides a reproducible baseline and public code and pretrained models for studying ILR-specific synthesis, distinguishing itself from prior synthesis work aimed at detection, segmentation, or autonomous driving.
• Shows one representation model can generalize across domains, relevant to universal embedding efforts, and that synthetic data can be merged with real data to improve results beyond either alone.
• Identifies a concrete weakness: generative models appear less effective for large objects with many details, based on the ROP results, which points at a target for future generative modeling work.

Real-world applications (domain areas used in the paper):

• Landmark recognition and retrieval, evaluated on R-Oxford, R-Paris, and GLDv2.
• Artwork recognition, evaluated on the MET dataset.
• Product retrieval from e-commerce images, evaluated on SOP.
• Everyday and multi-domain object retrieval, evaluated on INSTRE and mini-ILIAS, with the pipeline applicable to further domains such as fashion.

Industry relevance: the pipeline is designed to be efficient, requiring approximately 2.5 hours of training on an A100 for the default configuration, and it is adaptable because it leverages rapid advances in both LLMs and GDMs. This makes it relevant for teams that need domain-specific retrieval models but lack annotated instance-level data.

Future Directions

• Closing the gap on large, detail-rich objects: ROP performance is consistently better with real images than synthetic ones, which the authors attribute to potential generative-model shortcomings for large objects with many details; improving generation quality for such objects is a natural next step.

• Better viewpoint coverage: The pipeline's images of a class depict the object under different backgrounds but similar viewpoints that vary only due to padding. The authors mention exploring synthesized viewpoints by rendering multiple angles of GDM-generated instances and applying background generation to each rendered view, with details deferred to the supplementary material.

• Choosing and scaling the data recipe: The paper reports that 5K generic instances already help and that performance saturates between 10K and 20K, so the optimal budget of categories (C), instances per category (K), and backgrounds (N) remains an open question, with the paper noting an ablation over C and K in the supplementary material.

• Extending the mixing strategy: Since mixtures of real and synthetic data perform best but the ideal proportion varies by domain, with products differing from the rest, determining how to weight real versus synthetic data per domain is an unresolved question.

Target Audience

Researchers and practitioners working on image retrieval, instance-level recognition, and fine-grained representation learning who are interested in synthetic data generation; engineers who need to build domain-specific retrieval models without large annotated datasets; and readers following the integration of LLMs and generative diffusion models into computer vision training pipelines.

Authors’ abstract

Instance-level recognition (ILR) focuses on identifying individual objects rather than broad categories, offering the highest granularity in image classification. However, this fine-grained nature makes creating large-scale annotated datasets challenging, limiting ILR's real-world applicability across domains. To overcome this, we introduce a novel approach that synthetically generates diverse object instances from multiple domains under varied conditions and backgrounds, forming a large-scale training set. Unlike prior work on automatic data synthesis, our method is the first to address ILR-specific challenges without relying on any real images. Fine-tuning foundation vision models on the generated data significantly improves retrieval performance across seven ILR benchmarks spanning multiple domains. Our approach offers a new, efficient, and effective alternative to extensive data collection and curation, introducing a new ILR paradigm where the only input is the names of the target domains, unlocking a wide range of real-world applications. The code and pretrained models are publicly available at https://github.com/yankungou/ILGen.