Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

AI summary

Overview

Research area: Computer Vision / multimodal large language models (MLLMs), specifically reinforcement-learning post-training for Knowledge-Based Visual Question Answering (KB-VQA) with retrieval-augmented generation.

Technical level: Advanced. The paper assumes familiarity with policy-gradient reinforcement learning (GRPO/DAPO-style objectives), retrieval-augmented generation, and multimodal LLM post-training.

Scope: The paper proposes Wiki-R1, a curriculum reinforcement learning framework that generates training data of controllable difficulty and samples informative examples to adapt an MLLM to noisy, encyclopedic retrieval for KB-VQA, evaluated on Encyclopedic VQA and InfoSeek.

What This Paper Is About

Knowledge-Based VQA asks a model to answer a question about an image using external knowledge, typically retrieved from a large encyclopedic knowledge base such as Wikipedia. Because retrieval is inherently noisy and the retrieved material is structured and encyclopedic — unlike the data MLLMs see during pretraining — there is a large distribution gap between pretraining and this target task, which makes reinforcement learning post-training produce mostly useless (zero-advantage) training signals. Wiki-R1's goal is to close that gap by generating a sequence of training distributions of increasing difficulty, aligned with the model's evolving capability, and by selecting samples that actually produce learning signal.

Key Contributions

1. Wiki-R1 framework: A data-generation-based curriculum reinforcement learning framework that incentivizes MLLM reasoning for KB-VQA through a coupled data curriculum and sampling curriculum.
2. Controllable curriculum data generation: Rather than selecting from a fixed dataset, the method manipulates the retrieval system (number of retrieved candidates and whether the ground-truth snippet is enforced) to construct a hierarchy of training distributions, with difficulty adaptively adjusted based on the model's observed training accuracy.
3. Curriculum sampling with observation propagation: A sampling schedule that prioritizes samples likely to yield non-zero advantage, combined with a label-propagation graph over VQA samples (edge weights from similarity of associated knowledge base articles) that propagates sparse observed rewards to unobserved samples for difficulty estimation.
4. Empirical validation: New state-of-the-art results on two KB-VQA benchmarks and strong zero-shot transfer to ViQuAE, using a training set of only 40k examples.

Main Findings

• Sparse-reward diagnosis: Applying DAPO to KB-VQA, the authors observe that over 80% of samples exhibit zero advantage during training and overall training accuracy stays low at around 10%. Using ground-truth retrieval alleviates both problems, indicating retrieval noise is a major contributor.
• State-of-the-art on Encyclopedic VQA: Wiki-R1 improves accuracy from 35.5% (previous best, ReflectiVA with EVA-CLIP-8B and textual retrieval) to 37.1% (Wiki-R1 7B). The 3B variant reaches 35.9%.
• State-of-the-art on InfoSeek: Wiki-R1 improves accuracy from 40.1% (previous best, ReflectiVA with EVA-CLIP-8B and textual retrieval) to 44.1% (Wiki-R1 7B); the 3B variant reaches 42.2%.
• Strong performance on unseen splits: On the challenging InfoSeek Unseen-Question split, Wiki-R1 7B attains 47.8% — higher than its own InfoSeek overall accuracy of 44.1% — and on Unseen-Entity 42.3%; the 3B variant reaches 46.0% and 40.3% respectively.
• Single retrieval system robustness: Wiki-R1 uses one retrieval system (EVA-CLIP-8B + ColBERT V2, visual + textual) across both benchmarks, whereas prior methods such as ReflectiVA are highly sensitive to retrieval mode (e.g., 35.5 on EVQA with visual retrieval but 28.3 on InfoSeek with visual retrieval versus 40.1 with textual retrieval).
• Zero-shot transfer to ViQuAE: Wiki-R1 3B achieves 53.8 F1 and 48.6 Exact Match; Wiki-R1 7B achieves 55.6 F1 and 50.3 EM, surpassing the RC semi-oracle configuration (44.10 F1, 40.32 EM) and MLLM baselines such as ReflectiVA (23.2 F1, 38.1 EM).
• Oracle-entity setting: When only results from the ground-truth Wikipedia entity are provided, Wiki-R1 with Qwen-2.5-3B reaches 68.5 on EVQA Single-hop, 64.0 Unseen-Q, 65.9 Unseen-E, 65.3 overall on InfoSeek; the 7B variant reaches 69.2, 65.5, 69.5, 68.2. By comparison, ReflectiVA reaches 75.2 on EVQA but 57.8 / 57.4 / 57.6 on InfoSeek.
• Ablation results (Qwen-2.5-VL 3B): Zero-shot gives 18.6 EVQA Single-hop / 18.8 overall and 19.6 InfoSeek overall. SFT gives 21.6 / 25.1 on EVQA and 29.5 InfoSeek overall. DAPO gives 35.9 / 31.4 and 41.5. Adding the data curriculum raises these to 39.4 / 34.5 on EVQA and 43.0 InfoSeek overall. Adding curriculum sampling without observation propagation degrades performance (36.4 / 32.1 and 40.0), which the authors attribute to sparse observations. The full framework reaches 40.4 / 35.9 on EVQA and 42.2 InfoSeek overall.
• Observation propagation improves efficiency: Compared with vanilla DAPO and curriculum sampling alone, adding observation propagation significantly decreases the number of skipped trajectories (trajectories with zero advantage and no training signal).
• Training dynamics: DAPO improves rapidly in the first ~100 iterations but EVQA performance degrades as training proceeds, which the authors attribute to overfitting on the relatively easier InfoSeek data. Wiki-R1 improves stably on both benchmarks, with best performance emerging when training reaches the highest curriculum difficulty level.
• Similarity measure for label propagation: TF-IDF yields 40.4 EVQA Single-hop, 35.9 EVQA overall, 46.0 Unseen-Q / 40.3 Unseen-E / 42.2 InfoSeek overall (average 39.1); a Sentence Transformer yields 41.1, 36.3, 46.2 / 41.3 / 43.0 (average 40.7), showing the framework works with semantically richer representations.

Methodology in Plain English

The starting point is a pretrained multimodal LLM (Qwen2.5-VL, either 3B or 7B) that must answer questions using retrieved Wikipedia-style passages. Instead of training it directly on the hard target task, the authors build a ladder.

Step 1 — Generate data of controllable difficulty. They control the retriever through a discrete "gap level" g from 0 to G. At the easiest level (g = 0), the retriever returns k = 1 candidate and the ground-truth snippet is enforced (γ = 1), so the model sees clean, pretraining-like input. At intermediate levels, the number of retrieved candidates grows with g while the ground truth is still enforced, adding noise. At the hardest level (g = G), the ground truth is no longer enforced (γ = 0) and k = G − 1 candidates are retrieved, matching the real inference-time distribution. The maximum gap G is set to 6.

Step 2 — Move up the ladder based on performance. A sliding window of the most recent w = 300 samples gives a moving average of training accuracy. When this exceeds a threshold τ = 0.55, the gap level is promoted (g ↦ g + 1) and the stored observations are reset, so the model only faces harder data after mastering the current level.

Step 3 — Sample the most useful examples. Prior work indicates samples with training accuracy near 0.5 give the strongest gradient signal, so sampling follows a Gaussian centered at the historical mean training accuracy of 0.5.

Step 4 — Fill in missing difficulty estimates. On-policy rewards are sparse. The authors build a label-propagation graph over VQA samples, where edge weights reflect the similarity of the associated knowledge base articles, and propagate observed accuracies to unobserved samples. This lets the sampler estimate expected accuracy for examples it has not yet tried.

Implementation: Training uses the VERL framework with the DAPO algorithm, a rule-based binary reward (1 for exact match with the ground truth, 0 otherwise), learning rate 1e-6, and 4 rollouts per sample. Training data is entity-balanced: 20k examples from Encyclopedic VQA plus 20k from InfoSeek, for 40k total. Retrieval fuses EVA-CLIP 8B visual similarity with ColBERT V2 textual relevance (article chunks of size 256) via a weighted sum with λ = 0.985 for EVQA and λ = 0.997 for InfoSeek. Training takes 9 hours for the 3B variant and 12 hours for the 7B variant on 4 A100 GPUs.

Why This Matters

Impact on research: The paper shows that the dominant failure mode of RL post-training on KB-VQA is a distributional gap and sparse rewards, and that this can be addressed by generating (not merely selecting) data at controlled difficulty. It also demonstrates that the utility of curriculum sampling depends critically on estimating difficulty for unobserved samples — a point that prior selection-based curricula in multimodal RAG do not address. The results suggest that curriculum design, rather than raw data scale, drives gains: the training set here is only 40k examples, far smaller than baseline training sets.

Real-world applications (derived from the KB-VQA task the paper addresses):

• Question answering over large encyclopedic or enterprise knowledge bases paired with images, where retrieval is imperfect.
• Assistive and educational tools that answer detailed questions about specific entities such as species, landmarks, artworks or products.
• Multimodal assistants in e-commerce or catalog settings that must reason over structured product/entity databases plus user-supplied images.
• Domain-adaptive assistants for specialized image collections (e.g., biodiversity or cultural-heritage archives) where queries require facts beyond what is visible in the image.

Industry relevance: The framework shows strong results from small open-source backbones (3B and 7B Qwen2.5-VL) trained for 9–12 hours on 4 A100 GPUs with a 40k-example dataset, which is a modest compute and data budget compared with training retrievers or larger models. That makes the approach attractive for organizations that need domain-adaptive multimodal reasoning over a private knowledge base without full-scale retriever retraining.

Future Directions

• More fully controllable data generation. The authors state as a limitation that manipulating the retrieval system provides only a partial means of controlling the gap between pretraining and target distributions, rather than a fully controllable generation process; they flag this as a promising research direction.
• Extending the curriculum to other distributional gaps. The paper suggests the framework offers insights for domain-adaptive reasoning in retrieval-augmented multimodal settings beyond KB-VQA — for example, other downstream tasks with a pretraining-to-target gap.
• Better difficulty estimation and propagation. Since curriculum sampling alone degraded performance and only worked once observation propagation was added, improving how sparse on-policy signals are propagated (e.g., beyond the TF-IDF and Sentence Transformer similarity measures tested) is an open avenue.
• Managing cross-benchmark interference. DAPO's EVQA performance degraded over training, which the authors attribute to overfitting on the easier InfoSeek data; understanding and preventing such interference between mixed-difficulty data sources remains an open question.

Target Audience

Researchers and practitioners working on multimodal large language models, retrieval-augmented generation, and reinforcement learning post-training, particularly those adapting pretrained MLLMs to knowledge-intensive vision-language tasks with noisy retrieval. It is also relevant to engineers building domain-adaptive VQA or retrieval-augmented assistant systems who need effective training under limited data and compute. Readers without a background in RL policy optimization or RAG pipelines will find the method details demanding.

Authors’ abstract

Knowledge-Based Visual Question Answering (KB-VQA) requires models to answer questions about an image by integrating external knowledge, posing significant challenges due to noisy retrieval and the structured, encyclopedic nature of the knowledge base. These characteristics create a distributional gap from pretrained multimodal large language models (MLLMs), making effective reasoning and domain adaptation difficult in the post-training stage. In this work, we propose \textit{Wiki-R1}, a data-generation-based curriculum reinforcement learning framework that systematically incentivizes reasoning in MLLMs for KB-VQA. Wiki-R1 constructs a sequence of training distributions aligned with the model's evolving capability, bridging the gap from pretraining to the KB-VQA target distribution. We introduce \textit{controllable curriculum data generation}, which manipulates the retriever to produce samples at desired difficulty levels, and a \textit{curriculum sampling strategy} that selects informative samples likely to yield non-zero advantages during RL updates. Sample difficulty is estimated using observed rewards and propagated to unobserved samples to guide learning. Experiments on two KB-VQA benchmarks, Encyclopedic VQA and InfoSeek, demonstrate that Wiki-R1 achieves new state-of-the-art results, improving accuracy from 35.5\% to 37.1\% on Encyclopedic VQA and from 40.1\% to 44.1\% on InfoSeek. The project page is available at https://artanic30.github.io/project_pages/WikiR1/.