ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically knowledge-based visual question answering (KB-VQA) with retrieval-augmented generation and reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models, dense retrieval, chain-of-thought reasoning, and GRPO-style policy optimization.

Scope: The paper proposes ReAG, a retrieval-augmented pipeline that combines multi-level retrieval, a critic model for passage filtering, and a reinforcement-learning-trained generator that produces explicit reasoning traces before answering knowledge-intensive visual questions.

What This Paper Is About

Multimodal LLMs can describe images, but they fail on questions requiring specialized or encyclopedic knowledge that is underrepresented in pre-training data. The standard fix, retrieval-augmented generation, retrieves external Wikipedia-style documents, but those passages are often noisy, imprecise, or simply too numerous for the model to reason over reliably. ReAG's goal is to filter the retrieved content down to genuinely relevant evidence and then train the generator to reason explicitly over that evidence before giving an answer.

Key Contributions

1. A reasoning-augmented multimodal RAG architecture that pairs coarse-grained (whole-image) retrieval with fine-grained (cropped-region) retrieval and adds a critic model that classifies each retrieved passage as relevant or irrelevant. The critic is described as agnostic to the retrieval backbone, so it can be placed on top of any state-of-the-art retrieval engine.

2. A multi-stage training strategy for the generator in which supervised fine-tuning on reasoning traces serves only as a cold start, followed by a reinforcement learning stage inspired by GRPO, with rewards designed specifically for KB-VQA.

3. A rule-based reward scheme combining a task-specific accuracy reward (parsing predictions by question type: numerical or textual, single- or multi-answer) with a format reward that enforces a templated output structure delimited by <think>...</think> and <answer>...</answer> tokens added to the vocabulary.

4. State-of-the-art results on two KB-VQA benchmarks, Encyclopedic-VQA and InfoSeek, at both 3B and 7B generator scales, with interpretable reasoning traces grounded in retrieved evidence.

Main Findings

• Zero-shot MLLMs fail badly on knowledge-intensive questions. Without retrieval, BLIP-2 scores 12.6 on the E-VQA single-hop split and 12.5 on InfoSeek overall; LLaVA-v1.5-7B reaches 16.3 on E-VQA single-hop and 9.5 on InfoSeek; Qwen2.5-VL-7B reaches 23.6 and 23.7 respectively.

• Retrieval helps substantially. The paper notes InfoSeek overall accuracy rising from around 20% for zero-shot Qwen2.5-VL-7B to roughly 40% with retrieval-augmented methods such as mKG-RAG.

• ReAG sets new state of the art with EVA-CLIP-8B retrieval. With a Qwen2.5-VL-3B generator, ReAG scores 41.3 on E-VQA single-hop and 43.3 on InfoSeek overall (+7.6 and +4.3 over the reproduced ReflectiVA baseline). With Qwen2.5-VL-7B, it scores 44.9 on E-VQA single-hop and 47.2 on InfoSeek overall, a +4.8 and +3.3 improvement. The paper reports a +7.7 point E-VQA gain over ReflectiVA at the 3B scale and a +7.8 point improvement over VLM-PRF when using an InternVL3-8B backbone.

• ReAG also wins under the OMGM retrieval modality. With a Qwen2.5-VL-3B generator it gains +4.5 points on E-VQA and +5.1 on InfoSeek over ReflectiVA. At 7B it reaches 52.5 on E-VQA and 49.2 on InfoSeek, surpassing OMGM by 2.3 and 5.7 points respectively.

• ReAG leads in the oracle-document setting. When the ground-truth Wikipedia page is provided, ReAG reaches 79.0 on E-VQA single-hop and 54.2 on InfoSeek overall with Qwen2.5-VL-3B, and 81.5 and 59.7 with Qwen2.5-VL-7B. The 3B variant beats ReflectiVA by +6.1 points on E-VQA; the 7B variant beats ReflectiVA by +3.7 on InfoSeek and still exceeds it by +2.1 even when ReflectiVA uses the larger LLaVA-MORE-8B generator.

• Reasoning is robust when evidence is missing. In the analysis in Figure 3 (left), ReAG outperforms comparable-scale competitors both when the evidence passage is present and when it is absent, and stays closer to its own oracle upper bound than other methods.

• The critic reduces context noise. Figure 3 (right) shows ReAG passing 18.0% and 15.9% fewer passages to the generator than ReflectiVA based on LLaVA-MORE-8B and Qwen2.5-VL-3B respectively.

• Ablations confirm each component matters. At k=20 with the 3B generator, feeding all passages from the top-20 documents to a zero-shot model degrades performance to 19.2 on E-VQA single-hop and 10.1 on InfoSeek overall (versus 21.9 and 18.3 with no retrieval). Adding the critic lifts this to 38.0 and 27.0, adding the fine-grained retriever gives 40.2 and 27.1, SFT gives 39.3 and 37.5, adding reasoning traces gives 38.1 and 41.3, and the full SFT+reasoning+RL pipeline gives 41.3 and 43.3. The 7B ablation follows the same trend, with the full configuration reaching 44.9 and 47.2.

• Qualitative examples show ReAG correcting specific factual errors. On an InfoSeek bird-taxonomy question, zero-shot Qwen2.5-VL-7B answers "family Laridae" and ReflectiVA answers "Sterna", both marked incorrect, while ReAG answers "Thalasseus" correctly. On a bridge question, ReAG answers "A201" while ReflectiVA answers "Road". On a vehicle-brand question, ReAG answers "Isuzu" while ReflectiVA answers "Ford".

Methodology in Plain English

ReAG runs in four stages.

Retrieval. The whole query image is encoded with EVA-CLIP-8B and matched against the knowledge base to pull a first set of candidate passages (coarse-grained retrieval). Separately, the visual subject named in the question is extracted with spaCy and localized in the image with GroundingDINO; the resulting crop is re-encoded and used for a second retrieval pass (fine-grained retrieval). The top-k results from both stages, with k=20, are merged and ranked by relevance score, and all passages from the top-k documents form a noisy candidate pool. For InfoSeek the retrieval is image-to-text against page titles and summaries; for Encyclopedic-VQA it is image-to-image against images inside Wikipedia pages. Retrieval uses the FAISS library.

Filtering. A critic model, implemented as Qwen2.5-VL-3B fine-tuned for next-token prediction on an annotated subset of the ReflectiVA dataset, is given the image, question, and each candidate passage, and predicts "Yes" or "No" for relevance. Only passages predicted relevant above a probability threshold are kept.

Cold-start SFT. Each (image, question, passage, label) tuple is extended with a natural-language reasoning trace generated by an MLLM prompted with both the final answer and the relevance label. The generator is fine-tuned on these traces using a loss that balances answer tokens and reasoning tokens with α = 0.8, so answer tokens carry more weight. Output is forced into a templated format using special thinking and answer tokens.

Reinforcement learning. The generator is then optimized with a GRPO-inspired objective modified along DAPO lines: the KL divergence penalty is dropped to allow more exploration and save memory, and the loss is computed at the token level rather than averaged over sequences. Because updates are never off-policy, the importance ratio is always 1. Each batch contains 128 prompts with 8 completions per prompt, and rewards are normalized within each group to compute advantages. Rewards are a weighted sum of an accuracy reward (γ = 1.0) and a format reward (δ = 0.2).

Implementation specifics: the critic is trained for one epoch at a learning rate of 2×10⁻⁶ with a global batch size of 32. The generator uses AdamW with a learning rate of 2×10⁻⁶ and an effective batch size of 128 for SFT, and Adam with a learning rate of 1×10⁻⁶ for RL. The MLP adapter and LLM weights are updated while the vision encoder stays frozen. Both 3B and 7B Qwen2.5-VL generator variants are trained.

Evaluation uses BEM (a BERT-based matching score) for Encyclopedic-VQA and VQA accuracy plus relaxed accuracy for InfoSeek.

Why This Matters

Impact on research. The paper shows that retrieval quality and generator reasoning are separable problems that both need dedicated treatment: a critic handles noise at the retrieval end, while reinforcement learning with KB-VQA-specific rewards handles reasoning at the generation end. The finding that supervised fine-tuning alone is a weaker final configuration than SFT-as-cold-start plus RL is a useful data point for the growing body of work applying GRPO-style methods to multimodal models. The explicit reasoning traces also make the model's use of evidence inspectable rather than opaque.

Real-world applications:

• Encyclopedic and reference assistants that answer questions about specific landmarks, artworks, species, or products from images.
• E-commerce and catalog search, where a shopper photographs an item and needs brand, model, or attribute information not present in the pixels alone.
• Accessibility tools that let users query detailed factual properties of objects they photograph.
• Domain-specific inspection or maintenance workflows where model knowledge must be grounded in an external manual or database rather than internal parameters.

Industry relevance. The approach is designed to be backbone-agnostic on the retrieval side, which lowers integration cost for teams already running a retrieval stack. The reported gains at the 3B scale, where ReAG reaches 41.3 on E-VQA single-hop and 43.3 on InfoSeek overall, suggest that careful filtering and reasoning training can partly substitute for a larger generator, which matters for deployment cost.

Future Directions

• Extending the pipeline beyond Wikipedia-derived knowledge bases to domains where documents are more heterogeneous or where multimodal retrieval is less well studied.
• Investigating critic models that handle passage-level granularity beyond binary relevant/irrelevant classification, since the current design keeps or discards whole passages.
• Exploring whether the reasoning traces can be used to improve retrieval itself, closing the loop between generation and the retriever rather than treating retrieval as a fixed upstream stage.
• Testing scaling behavior beyond 3B and 7B generators, and evaluating whether the observed convergence toward oracle upper bounds continues at larger scales.

Target Audience

Researchers and practitioners working on multimodal retrieval-augmented generation, knowledge-based VQA, and reinforcement learning for MLLMs. It is most useful to readers already comfortable with dense retrieval pipelines and policy-gradient methods, but the problem framing and the ablation structure make the core ideas accessible to graduate students entering the KB-VQA area.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have shown impressive capabilities in jointly understanding text, images, and videos, often evaluated via Visual Question Answering (VQA). However, even state-of-the-art MLLMs struggle with domain-specific or knowledge-intensive queries, where relevant information is underrepresented in pre-training data. Knowledge-based VQA (KB-VQA) addresses this by retrieving external documents to condition answer generation, but current retrieval-augmented approaches suffer from low precision, noisy passages, and limited reasoning. To address this, we propose ReAG, a novel Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages, ensuring high-quality additional context. The model follows a multi-stage training strategy leveraging reinforcement learning to enhance reasoning over retrieved content, while supervised fine-tuning serves only as a cold start. Extensive experiments on Encyclopedic-VQA and InfoSeek demonstrate that ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.