Instance-Level Composed Image Retrieval

AI summary

Overview

Research area: Computer Vision — composed image retrieval (CIR), a retrieval setting where the query combines an image and a text and results must match both.

Technical level: Intermediate. The paper assumes familiarity with vision-and-language models (CLIP), embedding spaces, dot-product retrieval, and retrieval metrics such as mAP, but its central ideas are described conceptually and can be followed without deep mathematical background.

Scope: The paper introduces i-CIR, a new instance-level composed image retrieval benchmark built from LAION images, and basic, a training-free method that fuses image-to-image and text-to-image similarities from a pre-trained VLM to set new state of the art results on i-CIR and on four existing class-level benchmarks.

What This Paper Is About

Composed image retrieval is held back by the lack of good data, both for training and for evaluation: existing datasets are often built by taking two automatically selected similar images and describing their differences in text, which produces ambiguous queries, weak negatives, and text queries so specific that the image becomes redundant. This paper builds an evaluation dataset, i-CIR, whose class definition is instance-level — the goal is to retrieve images of the same particular object as the visual query (for example the Temple of Poseidon) transformed according to a text query (for example "during sunset"), rather than merely another image of the same semantic category. To avoid the corresponding training-data problem, the paper also proposes basic, a training-free baseline that combines image and text similarity from off-the-shelf VLMs and outperforms trained and zero-shot CIR competitors.

Key Contributions

1. i-CIR, a new instance-level evaluation dataset. It contains 202 object instances and 750K images in total, with 1,883 composed queries, and explicitly includes three curated negative types: visual hard negatives (same or similar object but not matching the text), textual hard negatives (matching the text semantics but a different instance), and composed hard negatives (nearly matching both).
2. basic, a training-free method. It separately estimates image-to-image and text-to-image similarity and performs a late fusion that upweights images satisfying both query parts while downweighting those that are similar to only one, with no database embedding updates.
3. Simple components that improve each similarity. Centering for bias removal, projection onto a semantic subspace derived from text features guided by an object corpus and a stylistic corpus, min-based score normalization, text query contextualization, optional image query expansion, and a Harris-corner-inspired fusion regularizer.
4. State-of-the-art results across five benchmarks. basic leads on the proposed i-CIR as well as on the class-level datasets ImageNet-R, NICO++, MiniDomainNet (MiniDN), and LTLL.

Main Findings

• basic beats all competitors on i-CIR. Average mAP is 31.64 for basic and 34.35 for basic without query expansion (basic†), against 28.50 for CoVR-2, 27.35 for MagicLens, 19.90 for SEARLE, 18.66 for CIReVL, 18.03 for WeiCom, 19.36 for Pic2Word, and 17.48 for the Text × Image baseline. Results are reported as macro-mAP over instances.
• basic also leads on every class-level dataset. ImageNet-R: 32.13 (basic) and 27.54 (basic†) versus 29.91 for FreeDom. NICO++: 31.65 and 28.90 versus 26.10 for FreeDom. MiniDN: 39.58 and 35.75 versus 37.27 for FreeDom. LTLL: 41.38 and 38.22 versus 33.24 for FreeDom.
• Query expansion helps class-level datasets but hurts i-CIR. The only datasets where basic† exceeds basic are i-CIR (34.35 versus 31.64), and the paper notes query expansion offers improvements particularly on category-level datasets while leading to a performance decrease in i-CIR.
• Strong per-category wins on i-CIR. basic ranks first in six of the eight visual categories, with the largest margins on fictional (47.8% versus 31.1% for SEARLE), mobility (45.8% versus 29.3% for MagicLens), and technology (30.6% versus 23.0% for Text × Image). It also leads on product (33.7% versus 26.7% for MagicLens), landmark (39.3% versus 35.0% for MagicLens), and art (38.0% versus 35.0% for MagicLens). The exceptions are fashion, where MagicLens leads at 25.6% versus 22.0%, and household, where MagicLens reaches 29.1% and basic is second at 22.4%.
• Textual-category results are similarly consistent. basic leads on projection (53.1% versus 31.1% for MagicLens), appearance (48.8% versus 36.8% for SEARLE), domain (39.3% versus 31.1% for MagicLens), viewpoint (47.8% versus 40.1% for MagicLens), and attribute (26.3% versus 24.1% for MagicLens), and is second on context (35.6% versus 36.4% for MagicLens) and addition (24.0% versus 28.2% for MagicLens).
• Centering gives a large early gain on i-CIR. In the ablation, adding centering on top of the Text × Image baseline raises i-CIR from 17.48% to 28.33%. The exception is LTLL, which the authors attribute to its narrow focus on landmarks.
• Min-based normalization is critical. The paper states its absence causes a significant drop; the ablation row without it gives 19.36 on LTLL against 41.38 for the full method.
• Text contextualization matters for nuanced language. Removing it drops i-CIR from 31.64% to 25.85%.
• Semantic projection drives much of the gain. The paper reports it accounts for the majority of the performance gain in many cases, and notes that some components depend on others (projection assumes centered features, the Harris step requires min-normalized scores).
• The negative corpus can be tuned to the application. Table 3 shows that using an application-related negative corpus improves results — for example MiniDN reaches 39.58 with its own corpus versus 39.38 with a generic one, and LTLL reaches 43.98 with its own corpus versus 41.24 with a generic one — though the paper notes that even generic corpora, such as ImageNet-1K class names, improve performance over omitting the stylistic corpus.
• i-CIR is compact but very hard. The Text × Image baseline scores 17.48% on i-CIR; the authors find that more than 40M distractor images would be needed for that baseline to reach a similarly low performance on unlabeled LAION negatives. This compares to about 3.7K database images per query on average and 750K database images across all queries. The authors note this is a lower bound because unlabeled LAION negatives inevitably include false negatives.
• i-CIR is genuinely compositional. Sweeping a mixing weight between text-only and image-only similarity for WeiCom, Text + Image, and Text × Image yields an average composition gain of +14.9 mAP on i-CIR.

Methodology in Plain English

The paper has two halves.

For the dataset, the authors start from human-defined instances such as the Temple of Poseidon and semantically plausible modifications such as "at sunset," explicitly avoiding implausible ones such as "with snow." They collect 2 to 5 high-quality seed images of the instance and of the instance under each modification, plus seed sentences describing the instance, other objects of the same category, rephrased modifications, and the instance under the modification. These seeds are used to retrieve neighbors from LAION, forming a candidate pool containing potential queries, positives, and hard negatives. Automated filtering removes low-resolution, watermarked, or duplicate content using perceptual hashing and resolution checks; annotators then inspect what remains to mark positives, select image queries from among the visual hard negatives, and the rest becomes that instance's database. Seed images are discarded so the dataset does not favor CLIP-based methods, and seed sentences avoid the exact wording of text queries. Each instance uses its own database for its n^v × n^t composed queries, which makes labeling scalable and error-free.

For the method, basic takes an image query and a text query encoded by a pre-trained VLM such as CLIP into a shared space (the experiments use CLIP ViT-L/14). It centers both query features by subtracting precomputed image and text means, computed on a large external dataset (LAION) and a predefined textual corpus respectively. Image features are then projected into a lower-dimensional subspace derived from text features: a weighted contrastive covariance matrix built from an object corpus (for example "building" or "dog") minus a stylistic corpus (for example "cartoon", "aerial view", "in a cloudy day") yields the top eigenvectors used as the projection. Text queries, which are often single words or fragments and therefore out of distribution for CLIP, are contextualized by combining them with terms from the object corpus to form caption-like phrases, embedding, centering, and averaging them. Image query expansion is an optional step that mixes the query feature with top-ranked database features. Finally, the two per-modality similarities are re-scaled so that the empirical dot-product minimum maps to 0 and 0 maps to 1, and fused by multiplication plus a Harris-inspired penalty term that suppresses items where only one modality is highly activated. Because the centering and projection can be rewritten so that all query-dependent computation happens on the query side, the stored database index needs no modification, and similarity search can be handled by existing libraries such as FAISS. Hyperparameters were fixed once on a small private development set, i-CIR dev: k = 250 PCA components, λ = 0.1 for the Harris criterion, and α = 0.2. The corpora for C+ and C- were generated with ChatGPT, and the normalization statistics were computed over a synthetic dataset built with Stable Diffusion from automatically created prompts. Performance is measured with mean Average Precision, with macro-mAP over instances for i-CIR.

Why This Matters

Impact on research. The paper argues that progress in CIR has been limited less by modeling than by data quality, and it supplies both a stricter evaluation benchmark and a strong training-free reference point. Because i-CIR requires instance-level identity plus textual modification, it cannot be solved by text alone or image alone, which the composition-gain analysis confirms. The dataset is deliberately compact to keep research practical while still being challenging, and basic gives future trained methods a competitive baseline that needs no training data at all.

Real-world applications.

• Searching personal or archival photo collections for the same building, person, product, or artwork captured under a different condition, season, or viewpoint.
• Product and fashion search where the user supplies a picture of a specific item and describes a desired variation.
• Media and creative asset retrieval, for example finding a specific landmark or character rendered in a requested style, at night, or from an aerial viewpoint.
• Location and cultural-heritage archives that need to track one particular site across time, weather, and depiction.

Industry relevance. The method requires no fine-tuning, no backpropagation, and no changes to the stored index, so centering and projection can be swapped per deployment or per user without rebuilding the database. That makes it attractive for large-scale image search systems

Authors’ abstract

The progress of composed image retrieval (CIR), a popular research direction in image retrieval, where a combined visual and textual query is used, is held back by the absence of high-quality training and evaluation data. We introduce a new evaluation dataset, i-CIR, which, unlike existing datasets, focuses on an instance-level class definition. The goal is to retrieve images that contain the same particular object as the visual query, presented under a variety of modifications defined by textual queries. Its design and curation process keep the dataset compact to facilitate future research, while maintaining its challenge-comparable to retrieval among more than 40M random distractors-through a semi-automated selection of hard negatives. To overcome the challenge of obtaining clean, diverse, and suitable training data, we leverage pre-trained vision-and-language models (VLMs) in a training-free approach called BASIC. The method separately estimates query-image-to-image and query-text-to-image similarities, performing late fusion to upweight images that satisfy both queries, while down-weighting those that exhibit high similarity with only one of the two. Each individual similarity is further improved by a set of components that are simple and intuitive. BASIC sets a new state of the art on i-CIR but also on existing CIR datasets that follow a semantic-level class definition. Project page: https://vrg.fel.cvut.cz/icir/.