Factorized MLLM Embeddings for Composed Image Retrieval

AI summary

Overview

Research area: Computer Vision — multimodal retrieval, specifically Composed Image Retrieval (CIR) with Multimodal Large Language Models (MLLMs). Technical level: Advanced (assumes familiarity with contrastive retrieval, InfoNCE, LoRA-style adapters, attention maps, and MLLM internals). Scope: The paper introduces FactEmbed, a training and inference scheme that makes an MLLM decompose a composed image-plus-text query into a small set of factorized latents that "imagine" the target image before forming a retrieval embedding, and shows state-of-the-art results on Refined-CIRR, Refined-FashionIQ, CIRR and FashionIQ.

What This Paper Is About

Composed Image Retrieval takes a reference image plus a text describing how the desired result should differ, and must return the matching target image. Recent MLLM-based systems simply push the whole composed query through the model and train the final [EOS] token contrastively against the target, which gives the model no explicit incentive to actually work out which visual attributes the text changes and which it preserves. FactEmbed's goal is to split that job in two: first let the model simulate the requested edit as an imagined target, then build the retrieval embedding on top of that decomposed imagination.

Key Contributions

1. Factorized latents as an information bottleneck. A set of K = 8 learnable tokens is prepended before the [EOS] token in both the query and target sequences, interacting with the multimodal input through the MLLM's ordinary causal attention. Their hidden states are the factorized representation; the MLLM's forward pass itself is left intact so kernel optimizations such as FlashAttention are preserved.

2. Target imagination as structural supervision. A lightweight decoder (one self-attention layer plus a 3-layer MLP) attached to layer ℓ = 14 of the 28-layer backbone reconstructs target patch features from a vision foundation model. Crucially, decoding is conditioned on a coarse 2 × 2 layout prior built from average-pooled ground-truth target features and added to the broadcast latents, so the latents are forced to carry only the fine-grained residual the prior cannot explain. Supervision is applied to both the query side (which never sees the target — this is the "imagination") and the target side, and requires no reasoning traces or dense annotations.

3. Turning imagination into a retrieval signal. A factor-level InfoNCE loss matches query and target latent sets using Sinkhorn optimal transport over pairwise cosine similarities, combined with an instance-level InfoNCE on the [EOS] embedding. A KL-divergence spatial distillation loss then copies the decoder's normalized alpha masks into the head-averaged latent-to-patch attention at the third layer from the top (ℓ_top = -3), so the decomposition survives into the layers actually used at retrieval time.

4. Complementary single-pass inference plus augmented post-training. One forward pass yields both the global [EOS] embedding and the latents. [EOS] retrieves from the full gallery; the mean-pooled latents (ũ) rerank only the top-100 shortlist, adding one vector per image. A final post-training stage uses MLLM-augmented triplets following ReCALL's pipeline, with the difference that the MLLM is conditioned on the original target image and asked what distinguishes a mined hard negative.

Main Findings

• Best overall results on the refined benchmarks. FactEmbed reaches R@1 of 68.99 and Subset R@1 of 87.45 on Refined-CIRR, and average R@10 of 71.13 and R@50 of 86.41 on Refined-FashionIQ.
• Clear margin over strong baselines. Against CSMCIR (which additionally uses synthetic captions for every gallery image) and ReCALL, the paper reports improvements of 2.7% and 2.3% in Recall@1 on Refined-CIRR, and 2.3% and 2.4% in average Recall@10 on Refined-FashionIQ.
• Best results on the original benchmarks too. On CIRR test: R@1 56.60, R@5 84.87, R@10 91.78, R@50 98.41, Subset R@1 81.86, Subset R@2 93.78, Subset R@3 97.61, Avg 83.37. On FashionIQ validation: Dress R@10 53.25 / R@50 74.67, Shirt 58.34 / 76.30, Top&Tee 61.55 / 79.65, average 57.71 / 76.87.
• Original benchmarks score much lower than refined ones. Excluding Qwen3-VL-Embedding 8B, the reported gap is at most 13.7 points in R@1 on CIRR and 15.6 points in average R@10 on FashionIQ; Qwen3-VL-Embedding 8B shows the largest gap at 15.1 and 16.7 points. The authors read this as evidence of ambiguous annotations and evaluation artifacts in the original datasets.
• Imagination alone is not enough, but it is a foundation. Adding only the target imagination objective (row b) improves R@10 on FashionIQ and Refined-FashionIQ by 1.00 and 1.16 and R@5 on CIRR by 0.99, but slightly lowers R@1 on CIRR — reconstruction structures the latents without providing a ranking signal.
• Every proposed component adds up. The full configuration raises Refined-CIRR R@1 from a 64.64 baseline to 68.99 and FashionIQ R@10 from 66.82 to 71.13 in the ablation table.
• Reranking gains come from the objectives, not just from fusing a vector. Adding efficient reranking on top of the trained model improves Avg. by 1.17 on CIRR and 0.97 on Refined-CIRR; applying the same reranking to latents trained with no auxiliary objective changes Avg. by only −0.37 and +0.24.
• Efficient reranking nearly matches full factor matching. Full optimal-transport matching of the latent set is best in raw R@1 (56.84 on CIRR, 69.20 on Refined-CIRR), but efficient reranking (56.60 / 68.99) narrows the gap to 0.24 and 0.21 while storing only two vectors per gallery image instead of K.
• [EOS] beats mean-pooling for the global embedding. Mean-pooled latents used directly as the global vector perform worst; even optimizing ũ with the instance-level objective in place of [EOS] still trails [EOS] in R@1, suggesting the instance objective pulls the latents toward a single embedding at the cost of fine-grained evidence.
• Layer 14 is the right place to attach the decoder. Among layers 7, 14 and 21, ℓ = 14 performs best, consistent with the trade-off between too little multimodal fusion and too few remaining layers to build a discriminative embedding.
• Optimal transport beats maximum similarity for factor matching on both benchmarks.
• The layout prior matters. Without it the latents collapse into coarse global semantics; a 1 × 1 prior is compared against the default 2 × 2, which guides the latents to isolate semantically editable parts such as object silhouettes.
• Augmentation and the framework are complementary. Post-training the baseline alone helps, but the full model still surpasses it by 2.91 and 2.57 in R@1 on CIRR and Refined-CIRR, and by 2.87 and 2.37 in R@10 on FashionIQ and Refined-FashionIQ.

Methodology in Plain English

The authors start from a standard recipe: fine-tune a Qwen2.5-VL-7B-Instruct model with DoRA adapters (rank 16, scaling factor 64, AdamW, cosine schedule with 10% warmup, batch size 512, bfloat16) so that a composed query lands near its target image in embedding space.

They change what sits inside the sequence. Eight extra learnable tokens are placed just before the [EOS] token, and these tokens are supposed to divide the scene into meaningful parts rather than each encoding the whole thing.

To force that division, they add a training pressure: from an intermediate layer, a small decoder tries to redraw the target image's patch features using only those eight tokens. Because the decoder is also handed a deliberately blurry 2 × 2 summary of the target's layout, the tokens cannot get credit for the coarse shape — they only reduce the error by capturing the sharp, localized details. On the query side the model must produce those details without ever seeing the target, which is the "imagination" step.

Reconstruction alone does not make a good search index, so two more pressures are added. A contrastive loss operates on the eight-token sets themselves, using optimal transport to allow flexible token-to-token correspondence between query and target. And the decoder's per-token spatial masks are distilled, via KL divergence, into the attention maps of the same tokens in a deeper layer, so the fine-grained factorization reaches the layers that actually produce the retrieval output.

Training is staged: first warm up the latents and decoder with the reconstruction loss alone, then train the backbone jointly with everything, then run a post-training stage on augmented triplets.

At test time, nothing extra is encoded. The [EOS] token gives one vector used to search the whole gallery; the eight latents are averaged into a single vector used only to re-score the top 100 hits. Both come from the same forward pass, so the accuracy boost costs one extra vector per image and no additional model calls.

Why This Matters

The paper argues that the compositional understanding already inside an MLLM is being wasted when it is trained only with contrastive supervision on a single output vector. Making the model explicitly represent the intended edit — and proving that this representation can be trained with a self-supervised reconstruction target rather than human-written reasoning traces — gives a practical route to better retrieval without extra inference passes or external LLMs at query time. The comparison to CCIN and FIRE-CIR, which call an external LLM or require verification linear in the number of candidates, makes the efficiency argument concrete.

Real-world applications:

• Fashion and e-commerce search, where shoppers upload an item and ask for "the same thing but longer sleeves" — the paper benchmarks heavily on FashionIQ and Refined-FashionIQ.
• Visual asset and stock-image discovery, where the desired result is easier to describe by relation to a reference than from scratch.
• Real-estate, automotive and marketplace listings, where a user points at one listing and asks for variations on a specific attribute.
• Assistive and accessibility search, letting a user express intent relative to an image they can point at.

Industry relevance: the design targets production constraints directly. Retrieval stays single-vector at gallery scale, the reranking overhead is independent of gallery size, and the factorized signal is obtained from the same forward pass rather than from an extra generative model or an extra LLM call — all of which matter for latency and index storage in deployed search systems.

Future Directions

• Rethinking the K = 8 / storage trade-off. Full factor matching is measurably better than the mean-pooled rerank, so a cheaper way to store and search the latent sets at gallery scale would capture the remaining 0.24 and 0.21 R@1.
• Extending factorization beyond images. The bottleneck-plus-layout-prior recipe is generic; whether the same structural supervision helps video retrieval, 3D scenes, or retrieval tasks where the "edit" is not spatial remains open.
• Combining with generative or world-model-based zero-shot CIR. Prior zero-shot work imagines the target with a proxy image generator or a world model; the paper's imagination lives inside the embedding model, and merging the two is an obvious follow-on.
• Making the evaluation itself more trustworthy. The large gap between original and refined benchmarks (up to 13.7 and 15.6 points for most methods, 15.1 and 16.7 for Qwen3-VL-Embedding 8B) suggests the community's numbers still depend heavily on annotation quality, and the paper leaves open how to benchmark compositional reasoning without that ambiguity.

Target Audience

Researchers and practitioners working on multimodal retrieval and embedding models, particularly those building or fine-tuning MLLM-based retrievers who want a training signal richer than plain contrastive loss without paying for extra inference. It is also relevant to vision researchers interested in slot- or latent-based decomposition (the paper connects to Perceiver, DETR and slot attention), and to engineers evaluating whether factor-level evidence is worth its storage cost in a production image-search system.

Authors’ abstract

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.