CoCo-IR: Contextual Composed Image Retrieval

AI summary

Overview

• Research area: Computer vision, specifically instruction-based image retrieval, multimodal embedding models, and Large Multimodal Models (LMMs).
• Technical level: Advanced. The paper assumes familiarity with contrastive learning (InfoNCE), transformer attention masking, embedding spaces, and LMM fine-tuning.
• Scope: The paper introduces a new multi-turn retrieval task (CoCo-IR), an LMM-based model (TIE) that generates context-aware image embeddings, an autonomous data engine, and benchmark results on single-turn and multi-turn retrieval.

What This Paper Is About

Existing composed image retrieval (CIR) systems take a source image plus one text instruction and return a target image, but they cannot handle the natural, iterative way people actually search: refining, exploring, or changing their mind across several turns. The paper defines Contextual Composed Image Retrieval (CoCo-IR), where each new instruction must be interpreted against the entire history of prior images and instructions. The goal is a single model that natively reads that interleaved image-and-text history and produces an embedding that tracks the cumulative transformation the user wants.

Key Contributions

1. A new task, CoCo-IR, which formulates instruction-based retrieval as a multi-turn dialogue where each instruction (T_t) is interpreted relative to the full history (H_t = (I_0, T_1, I_1, \dots, T_{t-1}, I_{t-1})), including at least one intermediate image reference per later turn.
2. The TIE model (Transformable Image Embedding), an LMM-based architecture that unifies multimodal comprehension and embedding generation end-to-end, using a dedicated ⟨EMB⟩ token as a global information bottleneck and a hybrid attention scheme (bidirectional attention within each turn, causal attention across turns).
3. A scalable, fully autonomous LMM-powered data engine that combines feature-based and metadata-based image-pair discovery, LMM instruction synthesis with self-reflection scoring, LMM-verified hard-negative mining, and LMM rewriting of later-turn instructions into context-aware dialogue.
4. State-of-the-art results on both single-turn and multi-turn retrieval, including 39.4 mAP@5 on CIRCO and 44.1 R@1 on 4-turn CoCo-IR dialogues, plus the release of a new CoCo-IR benchmark (project page: https://CoCo-IR.github.io).

Main Findings

• Single-turn CIR results: TIE-12B (Gemma3-12B backbone) reaches FIQ R@10 of 40.1, FIQ R@50 of 61.0, CIRR R@1 of 38.7, CIRR R@5 of 70.8, CIRCO mAP@5 of 39.4, and CIRCO mAP@10 of 40.2, giving it the best average rank (1.50) across the six metrics. TIE-4B reaches 40.1, 60.8, 37.7, 70.6, 37.8, and 38.4 (rank 2.75).
• Comparison to prior CIR methods: BGE-VL (CLIP-L) reaches 39.2 mAP@5 and 40.2 mAP@10 on CIRCO; MagicLens (CoCa-L) reaches 34.1 mAP@5; LDRE reaches 31.1; CIReVL 26.8; E5-V 19.1; CompoDiff 15.3; SEARLE 13.2. LinCIR has the highest FIQ R@10 at 45.1 but only 19.7 mAP@5 on CIRCO, which the authors attribute to linguistic alignment between its training corpus and FIQ.
• Multi-turn CoCo-IR results: TIE-12B achieves 1-turn R@1/R@5 of 76.3/90.4, 2-turn 54.8/82.1, 3-turn 44.6/76.6, and 4-turn 44.1/78.3. TIE-4B achieves 74.5/89.8, 48.3/79.5, 34.5/69.3, and 31.1/65.8 respectively.
• Single-turn baselines collapse on multi-turn: With the "first image + concatenated instructions" strategy, TIPS-SO400M, SigLIP-SO400M, and SigLIP2-SO400M drop to 0.0 R@1 by the 3rd and 4th turns, and MagicLens falls to 0.0 R@1 at 4 turns. The "last image + last instruction" strategy is similarly weak for the non-LMM baselines.
• Even a strong oracle does not close the gap: Using Gemini 2.5 Pro to summarize the whole history into one optimized instruction (the "Summarized Context" strategy), the best baseline (BGE-VL-MLLM-S1, LLaVA-NeXT-7B) reaches 60.8/79.4 at 1 turn but only 35.5/62.7 at 2 turns, 29.5/54.8 at 3 turns, and 28.2/53.9 at 4 turns — far below TIE-12B's 44.1 R@1 at 4 turns. E5-V with summarized context reaches 21.3 R@1 at 4 turns and MagicLens 10.7.
• Data engine is high quality: Human verification on random evaluation samples (representing 0.14%, 0.96%, 3.40%, and 7.65% of single-, two-, three-, and four-turn interactions) showed an average validity of 90.5%.
• Data efficiency: MagicLens (CoCa-L) uses 36.7M training pairs. TIE-4B trained on just 320K samples already surpasses it on all three single-turn benchmarks (FIQ R@10 38.5, CIRR R@1 34.8, CIRCO mAP@5 36.7), which the paper describes as more than a 100× reduction in data volume; the introduction separately states the model is trained with 14× fewer training samples than prior work.
• Scaling behavior: TIE-4B improves to 38.8/35.5/38.7 at 640K samples, 40.0/36.3/37.9 at 1.28M, and 40.1/37.7/37.8 at 2.56M. Gains begin to saturate beyond 1.28M samples for the 4B model, which the authors interpret as a model-capacity bottleneck, consistent with the gain from scaling to TIE-12B.
• Data-source ablation: Feature-based discovery alone gives FIQ R@10 31.6, CIRR R@1 36.9, CIRCO mAP@5 33.2, and CoCo-IR 1-turn R@1 65.7 (rank 4.50); metadata-based alone gives 34.2/36.6/32.7/60.7 (rank 4.75); mixing both gives 33.7/38.0/34.8/67.7 (rank 3.00). Adding LMM-verified hard negatives to the mixed set yields the best result: 40.1/37.7/37.8/75.4 (rank 1.50).
• Model design ablation: The full model combining bidirectional attention, single-side contrastive loss, and the special ⟨EMB⟩ token achieves the best rank (1.50) with 40.1 FIQ R@10, 37.7 CIRR R@1, 37.8 CIRCO mAP@5, and 75.4 CoCo-IR 1-turn R@1. Removing any one design degrades results; the paper reports each ablated variant at lower average rank (3.25, 2.25, and 3.00).
• Cost of multi-turn construction: Because of computation limits, the authors retain one hard negative per training sample and perform one iteration of model-guided hard-negative mining. Interaction chains are constructed up to four turns because longer sequences become increasingly sparse.

Methodology in Plain English

The authors first define multi-turn retrieval formally: the model sees the initial image, every previous instruction, and every previous retrieved image, and must retrieve the next correct image. Success is measured with a cumulative metric, (m)-turn Recall@k, which counts a query as successful only if the correct target appears in the top-k at every turn.

The model (TIE) is built on an existing large multimodal model rather than a standard retrieval encoder. Instead of predicting the next text token, the model is given a special ⟨EMB⟩ token at the end of each turn; the hidden state of that token, passed through a linear projection, becomes the query embedding. This is designed to avoid the bias of the common "last token" approach, which favors whatever was said most recently. Attention is bidirectional inside a turn (so image and text tokens in the same query can freely mix) and causal across turns (so the temporal order of the dialogue is preserved). Training uses an InfoNCE contrastive loss with cosine similarity and a fixed temperature, pulling the query embedding toward its target image and away from negatives in the batch, and the paper finds a single-side (non-symmetric) formulation works better than the symmetric CLIP-style loss.

Because no large annotated multi-turn dataset existed, the authors built an autonomous data engine: they find candidate image pairs both by clustering pretrained image features (with each image linked to its two nearest clusters and similarity range filtering) and by pairing images that co-occur on the same web page; an LMM (Gemini 2.5) writes a transformation instruction and then scores both the image pair's quality and its own instruction's quality, which is used as a filter. Candidate hard negatives — images close to the target but not satisfying the instruction — are verified by an LMM acting as a pairwise judge; for the evaluation set, negatives come from nearest neighbors in feature space, and for training, an initial TIE model proposes candidates that are then verified. Single-turn samples are chained into multi-turn dialogues by finding sequences where one sample's target is another's source, up to four turns, and an LMM rewrites the later instructions to refer naturally to earlier context ("the original image," "the second image"). Data is split at the image-cluster level so evaluation images are unseen, with stricter quality thresholds for the evaluation set.

Why This Matters

• Impact on research: It reframes composed image retrieval from a one-shot lookup into an interactive dialogue, defines an evaluation metric for that setting, and shows empirically that multi-turn visual context cannot be losslessly compressed into a single text prompt — motivating native multi-image, multi-turn retrieval models.
• Real-world applications:
  • Interactive e-commerce and fashion search, where a shopper refines a product query over several turns while referencing earlier results (the paper notes IRR and MAI are limited to the fashion domain, while CoCo-IR is open-domain).
  • Design and creative asset search, where a designer iterates on look-and-feel across related images before settling on a final asset.
  • Personal photo library and media archive exploration, where finding one image is a stepping stone to the next.
  • Exploratory search workflows such as real estate or interior design, where users progressively constrain a visual brief.
• Industry relevance: The data engine removes the expensive human annotation bottleneck for multi-turn retrieval data, so retrieval systems can be trained largely from web images and LMM supervision. The data-efficiency results — surpassing a 36.7M-pair baseline with 320K samples — make strong retrieval models practical for teams that cannot assemble web-scale labeled datasets.

Future Directions

• Longer and denser dialogues: The data engine caps interactions at four turns because longer sequences are increasingly sparse; extending to more turns is an open problem.
• More and better hard negatives: The authors retain only one hard negative per training sample and run one mining iteration due to computation limits, suggesting iterative mining or multiple negatives per sample could help.
• Evaluating realistic user adaptation: The benchmark fixes the trajectory so that any missed retrieval makes later context-dependent instructions ambiguous; simulating on-the-fly user adaptation instead of a static trajectory remains unsolved.
• Separating data limits from model limits: Gains saturate beyond 1.28M samples for the 4B model while scaling to 12B still helps, leaving open how far the data engine scales with larger models and whether it transfers to domains beyond open-domain web images.

Target Audience

Researchers and engineers working on multimodal retrieval, vision-language embedding models, and LMM fine-tuning; practitioners building interactive visual search or e-commerce retrieval systems; and benchmark designers interested in multi-turn evaluation protocols. Readers need a working background in contrastive learning and transformer architectures to follow the model and training details.

Authors’ abstract

Current instruction-based image retrieval systems are powerful but limited to single-turn interactions, failing to capture the iterative nature of complex, real-world visual searches. To overcome this limitation, we introduce Contextual Composed Image Retrieval (CoCo-IR), a novel task that enables users to progressively refine search results through interactions. We address this new task by proposing a new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR. Our model interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns. To fuel the model training without expensive human annotations, we develop a fully autonomous, scalable data engine that leverages LMMs to generate high-quality contextual retrieval data, and uses model-guided verification to mine challenging hard negatives. Extensive experiments demonstrate that our approach establishes new state-of-the-art performance: We achieve 39.4 mAP@5 on the challenging single-turn benchmark CIRCO; furthermore, on our new CoCo-IR benchmark, our model maintains robust performance with 44.1 R@1 on 4-turn dialogues, dramatically outperforming existing methods (28.2 4-turn R@1) that fail to handle multi-turn context. Project page: https://CoCo-IR.github.io.