ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval

AI summary

Overview

• Research area: Computer Vision, specifically composed image retrieval (CIR) using Multimodal Large Language Models (MLLMs).
• Technical level: Intermediate. The paper assumes familiarity with contrastive learning (InfoNCE), vision-language models, LoRA fine-tuning, and standard retrieval metrics such as Recall@K.
• Scope: The paper diagnoses a phenomenon it calls "Capability Degradation" in MLLM-based retrievers and proposes a model-agnostic diagnose–generate–refine framework (ReCALL) that is evaluated on the CIRR and FashionIQ benchmarks.

What This Paper Is About

Composed Image Retrieval takes a reference image plus a text describing how to modify it (for example, "make the dress half-sleeved") and must return the matching target image. When researchers adapt a generative MLLM into a single-embedding retrieval model, the model's native step-by-step reasoning quietly deteriorates, even though the same foundation model can answer the query correctly in its original generative VQA mode. ReCALL's goal is to detect and repair that lost fine-grained reasoning by using the foundation model itself to generate corrective supervision and then teaching the retriever from it.

Key Contributions

1. Identification of Capability Degradation: The authors name and quantify a previously overlooked problem — compressing a generative MLLM into a single-embedding discriminative retriever creates a paradigm conflict that damages fine-grained compositional reasoning.
2. The ReCALL framework: A model-agnostic diagnose–generate–refine pipeline that mines the retriever's own failure cases, prompts the foundation MLLM with Chain-of-Thought to build corrective instructions and triplets, and trains the retriever on them.
3. A Grouped Contrastive Refinement strategy: A structured batching scheme that places the original triplet and its corrected counterpart in the same micro-group, optimized with a dual objective combining InfoNCE loss and an in-group triplet margin loss.
4. State-of-the-art results on CIRR and FashionIQ: ReCALL reports improvements over both the baseline retriever and concurrent MLLM-based methods, plus ablations, backbone-generalization tests, and hyperparameter studies.

Main Findings

• Capability Degradation is measurable: On a subset of 1k instances where the foundation MLLM (ℱ) successfully retrieves the target (ℱ achieves 100% R@1), the fine-tuned baseline retriever (ℛ_base) achieves an R@1 of only 62.33% on CIRR and 55.80% on FashionIQ.
• CIRR test set results: ReCALL improves R@1 from 51.23% (ℛ_base) to 55.52%, a relative gain of +8.38%, and R_subset@1 from 77.57% to 81.49% (+5.06% relative). It surpasses the concurrent MLLM-based CIR-LVLM (53.64% R@1).
• FashionIQ validation set results: ReCALL reaches an average R@10 of 57.04% and R@50 of 76.42%, versus 53.23% and 74.37% for ℛ_base — relative gains of +7.16% and +2.76%. The largest category gain is +10.71% in average R@10 on the Dress category (46.80% to 51.81%).
• Self-guided mining beats random mining: Under an identical data budget, Random Mining (four seeds, sampled from the top-50 candidates of the frozen ℛ_base) moves R@10 from 53.23% to 53.80% ± 0.20, while Self-Guided mining reaches 57.04%. Random Mining's average R@50 is reported as 74.32% ± 0.06.
• Each component adds value: Removing components in the FashionIQ ablation gives R@10 of 53.23% (baseline), 55.41% (+ CoT-assisted Generation), 56.13% (+ VQA-Assisted Quality Control), and 57.04% (+ Grouped Contrastive Refinement). CG alone is an absolute gain of 2.18% in R@10.
• The problem persists in stronger models: With Qwen3-VL-8B, the baseline already reaches 55.93% R@1 on CIRR (versus 51.23% for Qwen2.5-VL-7B), yet ReCALL still lifts R@1 on CIRR to 57.09% and R@10 on FashionIQ to 57.60%.
• Performance scales with mined data: Without VQA quality control, increasing the mining hyperparameter K from 1 to 5 grows the training set from 13,351 to 57,125 samples, improving average R@10 from 55.27% to 56.07% and average R@50 from 75.70% to 76.29%.
• Hyperparameter preference: A grid search over λ ∈ {0.1, 0.2, 0.3, 0.4, 0.5} and m ∈ {0.05, 0.10, 0.20} finds λ = 0.3 and m = 0.05 optimal for FashionIQ, yielding the best average R@10 of 57.04%.
• Qualitative failure modes: ℛ_base retains coarse understanding (for example, "blue dress" or "wolf on snow") but misses specific constraints such as "half sleeved" or "facing the camera," while the foundation model answers correctly under VQA and ReCALL retrieves the correct targets.

Methodology in Plain English

ReCALL runs in four stages on a foundation MLLM (ℱ).

1. Baseline adaptation (Stage 1): Start from the foundation MLLM and fine-tune it into a retrieval model (ℛ_base) on composed image retrieval triplets using InfoNCE loss, so it can score query-target similarity. This step is what causes the capability degradation.
2. Diagnose (Stage 2): Run ℛ_base over the training set and discard queries it already ranks first. For the failure cases, collect the top-K images that were wrongly ranked above the true target. These "informative instances" are the ones that fooled the model and mark where its decision boundary is weak.
3. Generate (Stage 3): Prompt ℱ with Chain-of-Thought to explain, in two steps, why an informative instance does not match the query — first breaking the modification text into atomic intents and checking each against the reference and the informative image, then regenerating only the violated parts to produce a minimally edited instruction. This creates new corrective triplets. A VQA-based consistency check then filters out triplets whose answers are low-confidence or inconsistent.
4. Refine (Stage 4): Continue training starting from ℛ_base using grouped micro-batches that hold the original triplet and its corrected counterpart together, so the model must separate near-identical images using fine-grained text cues. The objective combines a global InfoNCE loss over the batch with an in-group triplet margin loss that pushes the informative instance away from the query, giving the final model ℛ_refine.

Setup details: The backbone is Qwen2.5-VL-7B, fine-tuned with LoRA at rank r = 16 on 8 NVIDIA H20 GPUs. FashionIQ uses a learning rate of 4×10⁻⁵, InfoNCE temperature τ = 0.03, global batch size 512, 200 optimization steps for Stage 1 and 250 for Stage 4, with λ = 0.30. CIRR uses a learning rate of 2×10⁻⁵, τ = 0.02, batch size 512, 300 and 350 steps for Stage 1 and Stage 4, with λ = 0.25. The triplet margin is m = 0.05 in both cases. Evaluation uses Recall@K, with R@10 and R@50 averaged over FashionIQ's three categories (Dress, Shirt, Top&Tee) and R@1/R@5/R@10/R@50 plus Recall_subset@K for K ∈ {1, 2, 3} on CIRR (a curated six-candidate subset).

Why This Matters

The paper reframes MLLM-based retrieval as a problem of preserving reasoning rather than only adding a retrieval head. Instead of indiscriminately synthesizing more training data, it shows that concentrating a fixed generative budget on a model's own failure cases is far more effective — a Random Mining baseline with the same data scale gained only 53.23% to 53.80% R@10, while self-guided mining reached 57.04%. It also demonstrates that the degradation persists even as foundation models get stronger (Qwen3-VL-8B), suggesting the issue is structural rather than a side effect of weak backbones. Reported experimental scope is CIRR and FashionIQ; no latency, throughput, or cost figures are reported in the provided content.

Real-world applications:

• E-commerce product search: the paper explicitly cites e-commerce as a target domain, and FashionIQ triplets are sourced from e-commerce websites.
• Fashion retail and styling: the FashionIQ categories (Dress, Shirt, Top&Tee) center on subtle attribute changes such as color, pattern, and style.
• Design workflows: the paper lists design among the domains where composed queries let users express more complex search intent.
• General visual search assistants: CIRR's open-domain, relation-heavy queries suggest use in everyday "find me an image like this but different" scenarios.

Industry relevance: the method is model-agnostic and built on LoRA fine-tuning of an open 7B/8B backbone, so it can be layered onto retriever systems that already exist rather than requiring a new architecture. Its diagnose-then-generate loop is also a reusable recipe for any discriminative adaptation of a generative model.

Future Directions

• Scaling the diagnose stage: The supplementary study varies K only from 1 to 5; how far the monotonic gain holds at much larger K, and whether quality filtering should scale with it, remains open.
• Reducing dependence on the foundation model's reliability: The pipeline trusts ℱ's CoT edits and VQA answers; the paper's own data-scale study deliberately omits VQA quality control, leaving the trade-off between filtering strength and data volume only partially explored.
• Broadening the benchmark set: Results are reported on CIRR and FashionIQ. Whether the same recalibration transfers to other retrieval tasks, other modalities, or longer multi-step modification instructions is not reported.
• Beyond CIR: The paper frames paradigm conflict as general to adapting generative MLLMs for discriminative tasks; extending the diagnose–generate–refine loop to detection, grounding, or ranking tasks is a natural next question.

Target Audience

Researchers and engineers working on multimodal retrieval, MLLM adaptation, and vision-language representation learning. It is most useful for those already comfortable with contrastive training and LoRA fine-tuning who want to understand why generative-to-discriminative adaptation loses reasoning ability, and how to measure and recover it. Practitioners building image search or fashion recommendation systems will find the pipeline design and the ablation evidence directly actionable, while the paper's diagnosis of capability degradation is relevant to anyone compressing generative models into embedding-based systems.

Authors’ abstract

Composed Image Retrieval (CIR) aims to retrieve target images based on a hybrid query comprising a reference image and a modification text. Early dual-tower Vision-Language Models (VLMs) struggle with cross-modality compositional reasoning required for this task. While adapting generative Multimodal Large Language Models (MLLMs) for retrieval offers a promising direction, we identify that this strategy overlooks a fundamental issue: compressing a generative MLLM into a single-embedding discriminative retriever triggers a paradigm conflict, which leads to Capability Degradation - the deterioration of native fine-grained reasoning after retrieval adaptation. To address this challenge, we propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline: First, we diagnose cognitive blind spots of the retriever via self-guided informative instance mining. Next, we generate corrective instructions and triplets by prompting the foundation MLLM and conduct quality control with VQA-based consistency filtering. Finally, we refine the retriever through continual training on these triplets with a grouped contrastive scheme, thereby internalizing fine-grained visual-semantic distinctions and realigning the discriminative embedding space of retriever with intrinsic compositional reasoning within the MLLM. Extensive experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance. Code is available at https://github.com/RemRico/Recall.