M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG

AI summary

Overview

Research area: Natural Language Processing / Vision-Language Models — specifically retrieval-augmented generation (RAG) for visual question answering across languages and cultures.

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, vision-language models, and embedding-based retrieval, but its core findings are conveyed through accuracy curves and two clearly defined metrics.

One-sentence scope: The paper introduces M4-RAG, a large-scale benchmark for multilingual, multicultural, multimodal retrieval-augmented VQA spanning 42 languages, 56 dialects and registers, 189 countries, and over 80,000 image-question pairs, and uses it to test when retrieval helps or hurts VLMs.

What This Paper Is About

Vision-language models answer visual questions using knowledge frozen at training time, which leaves them weak on culturally specific, region-specific information. Retrieval-augmented generation can supply that missing knowledge from external sources, but combining multilingual retrieval with multimodal (image-plus-text) retrieval is largely unexplored. The authors build M4-RAG to measure, at scale, whether retrieving culturally grounded documents actually improves VQA accuracy — and under what conditions it backfires.

Key Contributions

1. A massive-scale evaluation framework for multilingual multimodal RAG. It spans 42 languages with 56 regional dialects and registers across 189 countries, built on two cultural VQA datasets: CVQA and WorldCuisines. The authors state this is the first such framework, and Table 1 compares it against prior multilingual and multimodal datasets (MIRACL, MKQA, MRAG-Bench, Chart-MRAG Bench, and others), noting that M4-RAG offers broader linguistic coverage and explicitly annotates dialects.

2. A systematic study of retrieval strategies for VLM-based RAG. The work compares no retrieval, oracle context, text-based retrieval (oracle-query and caption-query variants), and multimodal retrieval, finding that naive text-based retrieval can degrade performance while multimodal retrieval provides more reliable gains — but does not consistently scale with model size.

3. Cross-lingual evaluation across 42 languages. The study measures how performance shifts when instructions and retrieved context are provided in non-English languages, finding significant degradation in both cases.

4. New retrieval corpora and open resources. The authors construct multilingual knowledge bases from April 2025 Wikipedia snapshots — 223,468 articles for WorldCuisines and 306,794 articles for CVQA — and release code, datasets, and evaluation protocols open-source.

Main Findings

• Baseline leaders: Gemma3 27B performs best in both CVQA and WorldCuisines at baseline, with accuracy of 74.34% and 66.20% respectively. With oracle context, Gemma3 27B performs best on CVQA, while Qwen-2.5-VL 72B performs best on WorldCuisines.

• Text-based retrieval hurts: Text-based retrieval performs worst among retrieval strategies — worse than the no-retrieval baseline across model sizes and datasets — indicating that converting images to text introduces noise that harms VLM performance. The B3 embedding model also shows comparatively lower gains than mmE5.

• RAG helps small models, stalls or hurts large ones: On CVQA, Gemma3 4B improves from 59.22% to 64.96% (+5.74%) with mmE5 retrieval, and Qwen2.5-VL 3B improves by +7.34%. In contrast, Gemma3 27B regresses from 74.34% to 72.59% with mmE5 RAG, and Qwen2.5-VL 72B shows a similar pattern. For larger models, the baseline eventually matches or surpasses multimodal RAG.

• Reasoning models are more robust: Reasoning VLMs consistently outperform non-reasoning models of comparable or larger size under RAG settings, maintaining retrieval gains longer as scale increases — though the slope of improvement still diminishes with scale.

• Correctness retention vs. correction rate diverge: Correctness retention (percentage of correct answers that stay correct with RAG) drops to 40–60% at retrieval relevance scores below 2.0, then converges tightly across all models toward 95–100% at high relevance. The correction rate (percentage of incorrect answers fixed by RAG) reaches 80–90% at high relevance but never saturates the same way, and model spread stays wide even at the highest relevance scores. Overturning a wrong answer is harder than preserving a correct one.

• Larger models have greater inertial priors: Qwen2.5-VL 72B and Gemma3 27B form the upper boundary of correctness retention and frequently the lower boundary of correction rate. Model scale increases reliance on parametric knowledge, reducing susceptibility to bad retrieval but also reducing willingness to adopt corrective retrieved evidence.

• Oracle–RAG gap widens with scale: On CVQA, oracle context achieves 94–99% accuracy, while the best RAG configurations reach only 64–74% for the largest models — a gap of 20–30%. On WorldCuisines, oracle performance reaches 74–80% while RAG variants plateau at 62–68%.

• English-centric bias in prompts: Switching from English to multilingual prompts consistently degrades performance across all resource levels — relatively mild for high-resource languages (mostly within -1% to -2%) but more severe for low-resource languages. Per-language analysis shows low-resource languages such as Amharic, Telugu, and Oromo often dropping by over 5–10%.

• Switching the retrieved context's language is far worse: With oracle context in the target language, performance deteriorates dramatically, with drops as large as -32.4% for Qwen2.5-VL 32B on CVQA and -28.8% for Pangea on CVQA for low-resource languages. Pangea, despite being trained explicitly on multilingual and multicultural Wikipedia data, is among the most severely affected.

• Smaller models hide the problem by code-switching: Smaller models show a lesser overall drop because they tend to code-switch to English even when prompted in a target language, whereas larger models attempt to respond fully in the target language and fail more dramatically.

Methodology in Plain English

The researchers gathered culturally focused image-question pairs from two existing datasets. CVQA contains more than 10,000 VQA pairs across 10 cultural categories, 30 countries, and 31 languages. WorldCuisines contains 60k VQA pairs parallel across 30 languages and dialects, centered on global cuisine and including adversarial prompts where provided context is deliberately misleading.

For retrieval, they built a document corpus from April 2025 Wikipedia snapshots. For each question they formed multiple query types — question-only, answer-only, and culturally enriched queries (for example, adding "Japanese cuisine" to a sushi-related item) — and retrieved the top 25 articles independently in English and in the corresponding target language so that non-English passages reflect culturally accurate terminology rather than direct translations. Articles were split by Wikipedia headings, cleaned of scripts, tables, and navigation elements, and deduplicated. The resulting corpora contain 223,468 articles for WorldCuisines and 306,794 for CVQA.

Models were tested under four main configurations, yielding six experimental variants per model. These were: (1) no retrieval baseline; (2) oracle context as an upper bound — for WorldCuisines this is the human-labeled food description, and for CVQA it is a caption generated by Qwen2.5-VL-72B-Instruct conditioned on the image, question, and human-annotated ground-truth answer; (3) text-based RAG using E5 as the multilingual dense retriever, in either an oracle-query variant or a caption-query variant using Qwen2.5-VL-72B-Instruct captions; and (4) multimodal RAG using the question and image jointly, tested with two embedding models, mmE5 (11B) and B3 (7B) from VLM2Vec. All RAG methods retrieve the top k=5 passages.

The VLMs evaluated were Gemma3 at 4B, 12B, and 27B; Qwen2.5-VL at 3B, 7B, 32B, and 72B; Qwen3-VL with reasoning at 4B, 8B, and 30B-A3B; and Pangea at 7B.

For evaluation, the authors used macro-averaged accuracy over multiple-choice answers. They validated their oracle captions with four annotators who rated 200 randomly sampled image-caption pairs on a 1–5 Likert scale — all samples received a score of 5 with full inter-annotator agreement. They also validated their VLM-as-a-judge scoring against humans on 100 samples per embedding model using five human raters and five reliability metrics: Fleiss' κ (0.6573 mmE5, 0.4273 B3, 0.5488 overall), Gwet's AC2 (0.7225, 0.5013, 0.6179), Krippendorff's α (0.6588, 0.4300, 0.5498), Conger's κ (0.6591, 0.4432, 0.5544), and Brennan-Prediger (0.7115, 0.4881, 0.6059) — indicating strong agreement for mmE5 and moderate agreement for B3.

For the cross-lingual study, prompts and oracle contexts were translated into each target language using Gemini-2.5-Flash, with all translations reviewed and validated by annotators. Languages were categorized using the Joshi et al. taxonomy from Class 0 to Class 5, and the benchmark includes coverage of low-resource languages in Classes 0–2 such as Oromo, Tigrinya, Sundanese, and Sinhala.

Why This Matters

The paper reframes the central question of RAG from "should we retrieve?" to "how do we make models actually integrate what is retrieved?" It shows that the bottleneck is not retrieval presence but retrieval utility — larger models can consume near-perfect context effectively (94–99% with oracle on CVQA) but fail to benefit from realistic retrieval, and sometimes perform worse than no retrieval at all. It also documents a persistent English-centric bias that survives even in models trained on multilingual and multicultural data.

Real-world applications:

• Cultural heritage and tourism assistants: Systems that identify regional dishes, flags, traditions, or landmarks, where the paper's example case of correctly identifying "Chitranna" versus "Yellow Rice" or "Biryani" illustrates concretely where retrieval quality changes answers.
• Multilingual customer support and commerce: Product and brand recognition across 42 languages, where the paper notes that incorrect retrieval actively misleads models into abandoning correct answers.
• Accessibility tools for low-resource language users: Since low-resource languages (Amharic, Telugu, Oromo, Tigrinya, Sundanese, Sinhala) suffer the largest degradation when context is provided in-language, these users are precisely the ones for whom current pipelines are least reliable.
• Regional content moderation and fact-checking: Systems that must ground claims in locally accurate rather than translated-approximate evidence.

Industry relevance: The findings suggest that model scaling and retrieval quality are not substitutable. A practical read is that throwing a larger VLM at a RAG pipeline may reduce, not increase, the benefit of retrieval, and that retriever choice matters — the gap between mmE5 and B3 in both accuracy and judge agreement (Fleiss' κ 0.6573 versus 0.4273) indicates that a weak retriever amplifies differences in how well models integrate context.

Future Directions

1. Model-aware retrieval strategies. The authors advocate optimizing for integration utility rather than query relevance alone, proposing directions such as joint retriever-VLM post-training or test-time adaptation.

2. Closing the oracle–RAG gap. With a 20–30% gap on CVQA and RAG plateaus of 62–68% versus 74–80% oracle on WorldCuisines, the authors identify retrieval quality as the primary bottleneck in multilingual multimodal RAG pipelines.

3. Cross-lingual evidence integration. The asymmetry between prompt switching (mild degradation) and context switching (drops up to -32.4%) suggests that cross-lingual evidence integration is a deeper bottleneck than instruction following, and the paper leaves open how to resolve it.

4. Reliability of evaluation. The divergence in judge agreement between mmE5 and B3 suggests an "understanding mismatch" where chunks receive higher localized scores despite inconsistent overall perception — an open question for how retrieval quality should be judged in multilingual settings.

Target Audience

Researchers and engineers building retrieval-augmented multimodal systems, especially those working on multilingual or culturally grounded applications. Also relevant to teams benchmarking VLMs, to practitioners deciding how much of a RAG pipeline's budget should go to retriever quality versus backbone model scale, and to anyone working on low-resource language coverage — since the paper's cross-lingual results are most severe for those languages.

Authors’ abstract

Vision-language models (VLMs) have achieved strong performance in visual question answering (VQA), yet they remain constrained by static training data. Retrieval-Augmented Generation (RAG) mitigates this limitation by enabling access to up-to-date, culturally grounded, and multilingual information; however, multilingual multimodal RAG remains largely underexplored. We introduce M4-RAG, a massive-scale benchmark spanning 42 languages, 56 regional dialects and registers, and 189 countries, comprising over 80,000 culturally diverse image-question pairs for evaluating retrieval-augmented VQA across languages and modalities. To balance realism with reproducibility, we build a controlled retrieval environment containing millions of carefully curated multilingual documents relevant to the query domains, approximating real-world retrieval conditions while ensuring consistent experimentation. Our systematic evaluation reveals that although RAG consistently benefits smaller VLMs, it fails to scale to larger models and often even degrades their performance, exposing a critical mismatch between model size and current retrieval effectiveness. Our cross-lingual evaluations also reveal significant performance degradation when prompts or retrieved context are provided in non-English languages. The code, datasets, and evaluation protocols for M4-RAG are available as open-source at https://github.com/davidanugraha/M4-RAG.