Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

AI summary

Overview

Research area: Information retrieval and retrieval-augmented generation (RAG), specifically page-level document retrieval in multilingual and visually rich (figure-heavy) settings.

Technical level: Intermediate. The paper assumes familiarity with standard retrieval terminology (BM25, dense embeddings, Top-K accuracy) but the argument itself is conceptual rather than mathematically heavy.

Scope in one sentence: The paper holds the retrieval algorithm fixed and varies only OCR/transcription and language-specific text normalization, showing that these upstream representation choices — not the retriever — explain much of the reported gap between classical lexical retrieval and modern multimodal systems.

What This Paper Is About

Recent benchmarks report that end-to-end multimodal retrievers substantially outperform classical lexical methods such as BM25 on multilingual documents and visually rich documents containing figures, tables, and complex layouts. The authors ask whether these benchmark gaps actually measure retrieval quality, or whether they largely measure the quality of the text representation being indexed. To test this, they keep the retrieval mechanism and evaluation protocol fixed while systematically swapping OCR/transcription pipelines and language-specific preprocessing, and observe how much of the apparent gap classical BM25 can recover.

Key Contributions

1. A controlled ablation design that separates transcription from retrieval. The authors vary only (i) the OCR/transcription used to build the page index and (ii) language-specific text processing, holding the retriever and evaluation protocol fixed on VisR-Bench.

2. Extending VisR-Bench transcription data with three additional OCR pipelines. They compare the dataset-provided Adobe Document Extract parser against EasyOCR, Mistral OCR 3, and Ministral 3B used as a small VLM transcriber, plus a set of language-specific normalization strategies.

3. Quantifying how much of the multilingual gap is representation-driven. Improving transcription quality and normalization alone recovers up to +8.9 Top-5 points for BM25 on average across multilingual datasets.

4. Identifying a distinct failure mode on figure-heavy pages. When figures have no textual or semantic description, retrieval degrades sharply; even coarse descriptions recover much of the loss, yielding gains of up to +31.1 Top-5 points.

5. A methodological call for decomposed benchmarks. The authors argue evaluation should separately measure transcription and retrieval capabilities so progress is correctly attributed.

Main Findings

• OCR and preprocessing reshape the multilingual leaderboard. In the full 15-language VisR-Bench table, BM25 with default transcription scores a macro-average of 51.08 Top-1 / 73.88 Top-5, while BM25 with the best per-language OCR and normalization configuration (reported as BM25*) reaches 57.28 Top-1 / 82.78 Top-5 — the +8.9 Top-5 improvement cited in the abstract.

• Text-based retrievers are sensitive to representation choices; multimodal retrievers are not. The relative standing of BM25 versus BM25* shifts substantially with OCR/preprocessing, whereas multimodal methods are described as invariant to OCR because they encode page images directly.

• Two distinct regimes across languages. For OCR-dominated languages (the paper names Arabic, Japanese, and Vietnamese), the choice of OCR model explains most of the performance variance and normalization cannot compensate for corrupted transcription. For preprocessing-dominated languages (Czech, Slovenian, Croatian), OCR choice is often secondary once a reasonable transcription exists, and lemmatization or morphology accounts for most of the gains.

• BM25 is highly sensitive to small representation changes. Enabling or disabling a single step — for example segmentation for Japanese or morphology for Arabic — can change Top-5 accuracy by over 10 points even when OCR is held fixed. The best configurations are typically simple, with one or two steps enabled.

• Visually rich pages show the largest recoverable gap. On VisR-Bench document-type breakdowns, BM25 with default Adobe Text Extract scores 46.12 Top-5 on figure documents versus 66.54 on table documents and 89.10 on text documents. With Ministral 3B transcription, BM25 reaches 77.24 Top-5 on figure documents — the paper's +31.1 point figure.

• Dense text retrievers benefit too. The controlled OCR ablations show gains for SBERT and BGE-M3 on figure-heavy pages when only transcription changes, indicating the representation bottleneck is not specific to lexical matching. For example, BGE-M3 goes from 56.80 Top-5 (default OCR) to 87.41 Top-5 (Ministral 3B) on figure documents.

• A hard boundary remains. Even with the best OCR and normalization, text-based retrievers stay well below state-of-the-art multimodal systems on visually grounded questions. The paper attributes this to evidence that is not recoverable as text, such as spatial relations, graphical trends, and non-textual encodings.

• A methodological warning. Because changing only OCR yields double-digit gains for a fixed retriever, the authors argue benchmark gaps should not be read as evidence of superior retrieval models; OCR and preprocessing should be treated as benchmark variables rather than hidden implementation details.

• A reported caveat. The paper notes that documents have different page-count distributions across languages, which should be considered when comparing results.

Methodology in Plain English

The study is built on VisR-Bench, a benchmark for retrieval-augmented question answering over long, visually rich documents. Each example has a document, a query, and a ground-truth evidence page, and the task is page-level retrieval: return the evidence page within the Top-K retrieved pages.

The core trick is experimental control. Instead of comparing models that differ in many ways at once, the authors fix the retriever and the evaluation protocol and change only how the page text is produced and cleaned up.

They compare four transcription pipelines: Adobe Document Extract (the dataset's default parser), EasyOCR (applied to rendered page images), Mistral OCR 3 (applied to page images), and Ministral 3B used as a small vision-language model with the prompt "Give me a markdown of what you see in the image. Reply only with the markdown content." Adobe Document Extract and EasyOCR perform text-only extraction, while Mistral OCR 3 also annotates figures and tables, and Ministral 3B is applied per image.

They also apply language-specific text processing to the multilingual runs: Snowball stemming for Romance and Germanic languages, spaCy lemmatization for highly inflected languages such as Czech, Slovenian, Croatian, and Finnish, CAMeL Tools morphological analysis for Arabic, and script-aware word segmentation for Japanese (Sudachi) and Vietnamese (pyvi). A minimal-processing baseline applies only lowercasing and NLTK tokenization. For each language, the best configuration is selected by Top-5 accuracy.

Retrievers span three families: sparse lexical retrieval (BM25), dense text retrievers (SBERT, BGE-large, BGE-M3, NV-Embed-v2), and multimodal retrievers (CLIP, SigLip, VisRAG, VLM2Vec, GME, and Col* methods including ColInternVL2, ColPhi, ColPali-v1.2, and ColQwen2 variants). Multimodal baseline numbers are reported from prior work; the authors' own evaluation focuses on varying OCR and preprocessing. Text-based retrievers index OCR transcriptions of pages, while multimodal methods encode page images directly.

Why This Matters

Impact on research. The paper challenges a widely repeated narrative that lexical retrieval is fundamentally inadequate for multilingual and visually rich documents. If OCR quality alone can close most of the gap, then benchmark leaderboards that conflate transcription quality with retrieval capability are misleading. The authors' proposal — decomposed benchmarks that separately measure transcription and retrieval — would let the field attribute progress correctly and focus effort where it actually helps.

Real-world applications:

• Enterprise document search. Organizations indexing scanned contracts, reports, and filings in many languages can get large gains from better OCR and language-appropriate normalization rather than replacing their retrieval stack.

• Multilingual customer support and knowledge bases. Languages like Arabic and Japanese, where segmentation and morphology matter enormously, benefit disproportionately from per-language preprocessing choices.

• Scientific and technical document retrieval. Figure-heavy material such as charts, plots, and diagrams is where default extraction fails most sharply; VLM-based transcription recovers much of the loss.

• Public-sector and regulatory archives. Visually rich forms, funding diagrams, and tabular documents are common, and the paper's worked examples (scaffolding product charts, storm drainage funding diagrams, sales tax growth charts) illustrate the kind of content where text extraction silently loses evidence.

Industry relevance. The finding shifts cost and effort toward data preparation. If transcription and normalization are first-class components of a retrieval system, then pipeline design, OCR model selection, and per-language text processing become primary engineering levers — and benchmark comparisons that omit these details become unreliable procurement signals.

Future Directions

1. Decomposed evaluation benchmarks. The authors explicitly call for benchmarks that measure transcription and retrieval separately, so that improvements can be attributed to the component responsible.

2. Query-aware and query-agnostic transcription prompting. Appendix D notes that the Ministral 3B prompt was deliberately simple, and suggests it would be interesting to optimize for lexical recall and query-agnostic retrieval effectiveness — for example, prompting for concise, noun-heavy, non-redundant descriptions.

3. Where representation improvements stop helping. The paper identifies a residual set of visually grounded questions where evidence is not recoverable as text (spatial relations, graphical trends, non-textual encodings). Determining exactly where that boundary lies remains open.

4. Language-specific pipeline design. Because a single global preprocessing pipeline is shown to be suboptimal, and because languages fall into OCR-dominated versus preprocessing-dominated regimes, deciding per-language configurations remains an open engineering and research question.

Target Audience

Researchers and practitioners working on retrieval-augmented generation, document AI, and multilingual search will get the most from this paper, particularly those who build or evaluate OCR-to-index pipelines and need to decide between upgrading the retriever or upgrading the transcription. It is also relevant to benchmark designers and to teams evaluating retrieval vendors, since it argues that headline benchmark gaps can be artifacts of hidden preprocessing choices. Readers need only a working understanding of standard retrieval evaluation (Top-K accuracy) and the BM25-versus-dense-retriever distinction.

Authors’ abstract

Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.