Let Multimodal Embedders Learn When to Augment Query via Adaptive Query Augmentation

AI summary

Overview

Research area: Multimodal Information Retrieval (Natural Language Processing / Information Systems), specifically multimodal embedding models for retrieval.

Technical level: Intermediate. The paper builds on contrastive embedding training and LLM-based query augmentation, so readers should be comfortable with embedding models, contrastive loss, and multimodal LLMs.

Scope (one sentence): The paper introduces M-Solomon, a multimodal embedder that learns to decide on its own whether a given query should be augmented before being embedded, rather than always augmenting or never augmenting.

What This Paper Is About

Recent LLM-based embedders can both write an augmentation for a query and embed it, which makes query representations more informative. But augmenting every query is slow, and for some queries the generated augmentation actually hurts retrieval. The authors show that this has never been tested in multimodal settings, so they build an embedder that learns when augmentation helps and when it does not.

Key Contributions

1. Adaptive query augmentation for multimodal embedding. M-Solomon, a universal multimodal embedder, learns to emit the prefix /augment followed by a synthetic augmentation when a query needs one, or the simple string /embed when it does not, deciding this at the start of generation.

2. Dataset-level division of training queries. The authors split the 20 MMEB datasets that have both training and test sets into 10 that "require augmentation" and 10 that do not, based on a pilot study comparing a model trained only with contrastive loss against one that always augments.

3. A synthesis pipeline using a Multimodal LLM. For queries requiring augmentation, a synthesis process using Qwen2.5-VL-72B-Instruct as a teacher model generates answer-style augmentations, following a prompt template that puts reasoning in <think> tags and the answer in <answer> tags (only the answer is kept).

4. Integration of generation and representation learning. The overall training objective combines an autoregressive generation loss with a contrastive embedding loss, so the same model both decides/generates augmentations and produces embeddings.

Main Findings

• M-Solomon beats both baselines on overall P@1. It reached 67.6 overall versus 66.1 for NoAug (no augmentation) and 67.4 for AlwaysAug (always augmentation), across 36 datasets (20 IND and 16 OOD).

• Adaptive selection is close to balanced. M-Solomon's /embed % was 55.1%, meaning it selected /embed and /augment roughly evenly, and its Confidence (CF) score was 93.1, indicating the choice was deliberate rather than random.

• Much faster than always augmenting. M-Solomon's Latency was 716 ms/query with 23.8 generated tokens, versus 1320 ms/query and 45.8 tokens for AlwaysAug — roughly half of both.

• Stronger out-of-distribution generalization. M-Solomon scored 65.4 on OOD versus 63.1 for NoAug and 64.7 for AlwaysAug.

• Dataset-level selection matters. M-Solomon-Half, which randomly augmented half of the queries in each dataset without dividing datasets, was worse overall (67.0) with higher latency (771 ms/query) and less confident augmentations (CF 67.6).

• Forcing a choice hurts. M-Solomon-/augment (always forcing augmentation) scored 67.3 overall and M-Solomon-/embed (always forcing embedding) scored 66.0, both below M-Solomon, and both showed abnormal generation halts caused by conflicts with queries of the opposite type.

• Per-dataset behavior reflects real adaptation. On FashionIQ, M-Solomon scored 26.7 P@1 versus 23.1 for NoAug and 21.1 for AlwaysAug, with /embed % of 91.0% — it correctly judged most FashionIQ queries did not need augmentation. On GQA it scored 68.1 versus 64.3 (AlwaysAug) and 61.5 (NoAug), with /embed % of only 8.3%. On ImageNet-R it scored 90.3 versus 88.5 and 85.3, with /embed % of 1.0%.

• Augmentation quality, not just quantity, differs. In a GQA example, AlwaysAug produced just "No" and retrieved an irrelevant document, while M-Solomon produced "No, the boat is on the left of the picture." and retrieved the relevant document. In an ImageNet-R example, AlwaysAug described the image as a hedgehog-like creature and retrieved "porcupine," while M-Solomon described a fluffy dog-like animal and retrieved "pomeranian," the relevant document.

• All trained models beat VLM2Vec. VLM2Vec scored 65.8 overall; NoAug, AlwaysAug, and M-Solomon all exceeded it, which the authors attribute to using hard negative documents and fewer samples. The paper states VLM2Vec used 662K samples without hard negative documents and 50K queries from each dataset.

Methodology in Plain English

The researchers started with a small pilot study on the FashionIQ dataset, training two models: one that only embeds queries using contrastive loss, and one that always generates an augmentation before embedding. The no-augmentation model retrieved the right image quickly, while the always-augment model misread the query and exaggerated "Paris" into Eiffel Tower-style designs, which led it astray.

Based on that, they split the MMEB training datasets at the dataset level. The 10 datasets where the no-augmentation model performed better or comparably became the "not requiring augmentation" group, and the other 10 became the "requiring augmentation" group. For the second group, they used a strong Multimodal LLM as a teacher to generate answer-style augmentations, treating the answer to a query as its augmentation because answers carry useful information.

Then they trained M-Solomon (built on Qwen2-VL-7B-Instruct) with two objectives at once: a contrastive loss that pulls the augmented query's embedding toward its positive document and away from hard negatives, and an autoregressive loss that teaches the model to output /augment plus a synthetic augmentation for queries that need one, or just /embed for the rest. At inference, the model first generates one of these tokens; if it picks /augment, the generated augmentation is appended to the query, and the augmented query is encoded — all in a single forward pass.

Training used 2.5K queries from each of the 20 datasets, totaling 50K samples, on a single node with 8×A100 80GB GPUs, with LoRA rank 16, image resolution 512×512, maximum token length 1800, one epoch, learning rate 2e-5, and hyper-parameters m=1, N=128, τ=0.02, α_rep=1.0, α_gen=0.1. The paper notes that contrastive loss on the original queries (in addition to augmented ones), as used in prior work, was found ineffective and was omitted.

Why This Matters

Impact on research: The paper opens a multimodal angle on a problem previously studied only for text, and shows that adaptive augmentation can beat both extremes (never augment, always augment) on accuracy and latency. It also offers a concrete recipe — dataset-level separation plus a teacher MLLM synthesis step — that others can reuse, and it identifies query-level decisions as the natural next step.

Real-world applications:

• Multimodal product search, where short or ambiguous shopper queries benefit from elaboration but others are better left alone.
• Image and video retrieval systems where embedding latency directly affects user experience and infrastructure cost.
• Visual question answering and classification pipelines that route queries through a retrieval-style embedding step.
• Large-scale content moderation or tagging systems (the paper's training set includes HatefulMemes) where fast, accurate embedding matters at scale.

Industry relevance: Halving latency while improving accuracy is directly valuable for production retrieval systems, where augmentation on every query is expensive. The paper comes from NC AI and is published at CIKM '25, an information retrieval venue, and the method is described as easily integrable into existing approaches rather than requiring a new architecture.

Future Directions

1. Query-level instead of dataset-level decisions. The authors state they will study methods to identify which queries require augmentation at the query level, since this reflects fine-grained information of each query.

2. A reasoning-based augmentation option. They plan to extend adaptive query augmentation with an additional option that performs reasoning-based query augmentation for reasoning-intensive embedding tasks such as BRIGHT and RAR-b.

3. Generalizing beyond the two-way /augment vs /embed choice. Since the framework already supports a binary decision, adding more modes (for example a reasoning mode) is a natural extension the paper explicitly names.

4. Robustness of the dataset split. The current split depends on a pilot study over 20 MMEB datasets; how stable those assignments are for new datasets or domains is left open.

Target Audience

This paper is most useful for NLP and information retrieval researchers and engineers working on embedding-based retrieval, especially those building multimodal search systems or fine-tuning multimodal LLMs. It also suits practitioners who want practical accuracy and latency gains without redesigning their retrieval stack, and students with some background in contrastive learning who want a clear example of joint embedding-and-generation training. The paper does not report training cost, wall-clock training time, or memory usage, so readers seeking those operational details will not find them here.

Authors’ abstract

Query augmentation makes queries more meaningful by appending further information to the queries to find relevant documents. Current studies have proposed Large Language Model (LLM)-based embedders, which learn representation for embedding and generation for query augmentation in a multi-task manner by leveraging the generative capabilities of LLM. During inference, these jointly trained embedders have conducted query augmentation followed by embedding, showing effective results. However, augmenting every query leads to substantial embedding latency and query augmentation can be detrimental to performance for some queries. Also, previous methods have not been explored in multimodal environments. To tackle these problems, we propose M-Solomon, a universal multimodal embedder that can adaptively determine when to augment queries. Our approach first divides the queries of the training datasets into two groups at the dataset level. One includes queries that require augmentation and the other includes queries that do not. Then, we introduces a synthesis process that generates appropriate augmentations for queries that require them by leveraging a powerful Multimodal LLM (MLLM). Next, we present adaptive query augmentation. Through this step, M-Solomon can conduct query augmentation only when necessary by learning to generate synthetic augmentations with the prefix /augment for queries that demand them and to generate the simple string /embed for others. Experimental results showed that M-Solomon not only surpassed the baseline without augmentation by a large margin but also outperformed the baseline that always used augmentation, providing much faster embedding latency.