Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generation

AI summary

Overview

Research area: Multimodal Retrieval-Augmented Generation (MRAG) for Multimodal Large Language Models (MLLMs), with a focus on adaptive retrieval control.

Technical level: Intermediate — readers should be comfortable with retrieval-augmented generation, vision-language models, and instruction tuning, but the paper's core ideas are explained without requiring deep implementation knowledge.

Scope: The paper proposes Windsock, a query-dependent module that decides whether to retrieve at all and which modality (visual or textual) to retrieve, plus DANCE, a noise-resistant instruction-tuning strategy, evaluated on WebQA and MultimodalQA.

What This Paper Is About

Existing multimodal RAG systems retrieve documents for every query, always pull from the same fixed modality, and often mishandle irrelevant retrieved content. This paper addresses three questions: when to retrieve, what modality to retrieve, and how to make an MLLM use retrieved information robustly. The goal is to cut unnecessary retrieval computation while improving the factual quality of generated answers.

Key Contributions

1. Windsock — a lightweight, query-dependent decision module built on a Flan-T5-Small backbone that performs three-way classification over queries into NA (no retrieval), Visual, or Textual retrieval strategies, and can be extended to additional retrieval types such as Hybrid.

2. DANCE (Dynamic Noise-Resistance) Instruction Tuning — an adaptive training strategy that scores a model's responses under visual versus textual retrieval and selects the modality where the model struggles most as the challenging case for fine-tuning, improving robustness to irrelevant retrieved documents.

3. A self-assessment data curation pipeline — a method that converts any question-answering dataset into an MRAG training set without proprietary models or human annotations, by generating responses under different retrieval strategies, scoring them against ground truth, and deriving both the optimal retrieval strategy for Windsock and the challenging modality for DANCE.

4. Extensive empirical validation across WebQA and MultimodalQA with LLaVA-v1.5 and Qwen2-VL generators, including efficiency measurements, ablations, noise benchmarks, and scaling studies.

Main Findings

• Overall gains: The abstract reports that the proposed method improves generation quality by 17.07% while reducing retrieval times by 8.95%.

• Best configuration: Combining Windsock with DANCE yields the best results across all metrics, measured with Qwen2-VL-7B-DANCE: WebQA Single 70.12, Multiple 59.32, All 62.36, and MultimodalQA F1 52.72, EM 48.20. This is an improvement of 16.04% on WebQA and 5.19% on MultimodalQA over the "w.o. Instruction Tuning" baseline, and 4.75% / 1.97% over SURf.

• Windsock alone helps: With Qwen2-VL-7B (no instruction tuning), Windsock reaches WebQA All 46.32 and MultimodalQA F1 46.90, versus Zero-Shot's 44.04 / 29.94 and Vanilla RAG's 45.29 / 34.37.

• No single modality wins: Using only visual retrieval (WebQA All 44.47 w.o. DANCE) or only textual retrieval (41.25 w.o. DANCE) does not consistently beat alternatives, while Windsock reaches 46.32 — evidence that different queries need different modalities.

• Efficiency: Windsock averages 0.36 s per query with DANCE (versus 0.79 s for Textual-only w.o. DANCE), while scoring higher. NA retrieval is fastest at 0.46 s w.o. DANCE but performs poorly.

• Skip rates: Windsock skips retrieval for 8.96% of WebQA queries; when 2,000 simpler MS-COCO validation queries are mixed in, the skip rate rises to 26.99%.

• Low overhead: The generator consumes 642.86 ms (96.94%) of total inference time; Windsock adds only 10.25 ms (1.83%).

• Ablations: Removing modality selection drops WebQA All from 46.32 to 43.89; a two-stage "w.o. One-stage" design drops it to 44.13.

• Instruction tuning strategy: DANCE beats "Easy" modality selection (59.97 vs 53.38 WebQA All) and "Random" (55.12).

• Noise rejection: On manually created noisy documents, Qwen2-VL-7B-DANCE scores 66.97 F1 versus SURf's 64.07. On fine-grained noise benchmarks built from 521/472 MDC/CMC samples, DANCE scores 68.9 (+3.4) on MDC and 59.5 (+1.3) on CMC.

• Ground truth documents: Given gold documents, Qwen2-VL-7B improves from 50.10 to 67.35 F1 with DANCE; LLaVA-v1.5-13B from 44.68 to 64.07.

• Data pipeline efficiency: Dataset construction takes 15.43 GPU hours for the proposed method versus 32.45 for SURf.

• General ability trade-off: On the MME benchmark, DANCE slightly lowers Qwen2-VL-7B's perception (1423.25 to 1394.05) and cognition (484.00 to 435.71) scores, but LLaVA-v1.5-13B's cognition rises from 294.64 to 309.64.

• Response length: Without DANCE, average response length is 16.2 tokens; with DANCE it drops to 9.8 tokens, which shortens autoregressive generation and lowers inference time.

• Retrieval size: Performance improves with top-k up to k = 3, with diminishing returns beyond that.

• Retriever choice: Marvel ANCE leads among tested retrievers (MRR@1 55.44%, Recall@5 73.95%), followed by VisualBGE (46.28%, 64.30%), with CLIP ViT-L-14-336 far lower (9.72%, 17.14%); VisualBGE is used by default.

• Windsock backbone: Flan-T5-Small scores 58.3 F1, beating DistilBERT-Base (55.5) and CLIP ViT-B/16 (49.2).

• Decision quality: Windsock as a decision module reaches EM 43.2 versus GPT-4o's 40.7 on the same role. Applied on top of ReflectiVA on E-VQA, it raises EM from 35.5 to 36.9 while skipping retrieval for 7.6% of queries (ReflectiVA: 0%).

• Extensibility: Extending to Hybrid retrieval yields 59.1 F1 (+1.1 over the best single strategy), and adding a pseudo-video modality yields 51.0 (+1.2).

Methodology in Plain English

The system works in three stages.

1. Decide the retrieval strategy. A small module called Windsock looks at the input query and picks one of three actions: answer directly with no retrieval (NA), retrieve from a visual knowledge base (Visual), or retrieve from a textual knowledge base (Textual). This is a single three-way classification, implemented with Flan-T5-Small, rather than a chain of separate decisions.

2. Build training data by self-assessment. For each question-answer pair in a dataset, the authors run the MLLM three ways — no retrieval, visual retrieval, and textual retrieval — and score each response against the ground truth using a task metric such as F1. The best-scoring strategy becomes Windsock's training label. This avoids expensive human or GPT-4 annotation.

3. Train the generator to resist noise. DANCE takes the same three scores and deliberately selects the modality where the model scored lower (randomly breaking ties) as the "challenging modality." The documents from that modality are likely irrelevant, so training on {question, retrieved documents, answer} pairs teaches the MLLM to ignore misleading content. The authors note they ignore the NA score during this dataset construction. Fine-tuning follows a standard instruction-tuning pipeline with default LoRA settings in LLaMA-Factory, with training epochs set to 1.

Setup details: Retrieval uses VBGE-base returning the top 3 documents. Windsock trains with AdamW at a learning rate of 5e-4, batch size 16, for 5 epochs with a linear scheduler and class-weighted cross-entropy. All training runs on 4 NVIDIA H100 GPUs and inference on 1 NVIDIA H100 GPU.

Why This Matters

Impact on research. The paper reframes multimodal RAG as a decision problem rather than a fixed pipeline. It shows that retrieval can actively hurt answer quality when the model already knows the answer, and it provides a labeling-free way to derive retrieval strategy supervision from any QA dataset. It also offers a cheaper alternative to GPT-4-based annotation and to sequential hard-example mining like SURf, cutting dataset construction from 32.45 to 15.43 GPU hours.

Real-world applications:

• Open-domain question answering over mixed image and text corpora, where some questions need a photo and others need a document.
• Assistive or search products where latency matters and skipping a retrieval step saves compute (Windsock's overhead is only 10.25 ms, 1.83% of pipeline time).
• Enterprise knowledge assistants that must stay accurate when a retriever returns wrong or contradictory documents — the setting DANCE's noise benchmarks target.
• Deployments where proprietary model APIs are unavailable for building training data, since the self-assessment pipeline needs no proprietary models.

Industry relevance. The modular design is explicitly described as integrable into both open-source and proprietary MLLMs, and Windsock is reported to outperform GPT-4o as a decision module (EM 43.2 vs 40.7) at a fraction of the cost. Efficiency gains matter directly for serving costs, and the paper's efficiency comparison includes real GPU-hour measurements.

Future Directions

1. Multimodal Windsock backbones. The current Windsock uses a text-only backbone (Flan-T5-Small), which the authors justify by noting that textual queries capture user intent, but they state the framework can be extended to multimodal backbones in future work.

2. More modalities and hybrid retrieval. The paper demonstrates Hybrid retrieval (59.1 F1) and a pseudo-video modality (51.0 F1) and notes that no existing RAG dataset supports more than three modalities, leaving genuinely multi-modal benchmarks as an open need.

3. Stronger retrievers. The authors use VisualBGE by default but show Marvel ANCE scores higher across all retrieval metrics, explicitly naming better retrievers as future work.

4. Recovering general MLLM ability. DANCE introduces a trade-off with MME perception and cognition scores, and the paper flags improving general ability as an open direction. Two error modes are also identified and left unresolved: Windsock picking the wrong modality, and the generator being distracted by noisy documents even when the right documents are retrieved.

Target Audience

Researchers and engineers working on retrieval-augmented generation, multimodal LLM systems, or vision-language question answering will benefit most. The paper is also useful for practitioners who need to reduce inference cost in RAG deployments, and for those building training data without access to proprietary annotation models. Readers new to RAG will find the three-challenge framing (when/what/how) an accessible entry point, though the results tables assume familiarity with F1, EM, MRR, and Recall metrics.

Authors’ abstract

Multimodal Retrieval-Augmented Generation (MRAG) has emerged as a promising method to generate factual and up-to-date responses of Multimodal Large Language Models (MLLMs) by incorporating non-parametric knowledge from external knowledge bases. However, existing MRAG approaches suffer from static retrieval strategies, inflexible modality selection, and suboptimal utilization of retrieved information, leading to three critical challenges: determining when to retrieve, what modality to incorporate, and how to utilize retrieved information effectively. To address these challenges, we introduce Windsock, a query-dependent module making decisions on retrieval necessity and modality selection, effectively reducing computational overhead and improving response quality. Additionally, we propose Dynamic Noise-Resistance (DANCE) Instruction Tuning, an adaptive training strategy that enhances MLLMs' ability to utilize retrieved information while maintaining robustness against noise. Moreover, we adopt a self-assessment approach leveraging knowledge within MLLMs to convert question-answering datasets to MRAG training datasets. Extensive experiments demonstrate that our proposed method significantly improves the generation quality by 17.07% while reducing 8.95% retrieval times.