VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

AI summary

Overview

Research area: Vision-based document information retrieval and multimodal retrieval benchmarking (page-image retrieval over long, visually rich documents).

Technical level: Intermediate. The benchmark design and motivation are accessible, but the evaluation section relies on dense retrieval metrics (nDCGAll@k, RecallAll@k), late-interaction scoring, and multi-stage retrieval settings.

Scope in one sentence: The paper introduces VL-DocIR, a page-level benchmark of 29,641 rendered documents, 388,548 page images, and 271,760 queries across 23 domains that requires systems to retrieve the complete set of evidence pages, including cases where evidence spans multiple pages or multiple documents.

What This Paper Is About

Existing visual document retrieval evaluations mostly use short documents with a single relevant page, so a system can score well while still failing on realistic long documents where the evidence for a question is spread across pages, mixed between text and visuals, or split across separate documents. The paper builds a benchmark that only counts a retrieval as successful when every required evidence page appears in the top-k results. It then uses that benchmark to measure how leading single-vector, multi-vector, OCR-based, text-based, and hybrid retrievers behave under this stricter, evidence-complete criterion.

Key Contributions

1. A large-scale evidence-complete retrieval benchmark: VL-DocIR contains 29,641 documents, 388,548 rendered page images, 271,760 queries, and 23 domains, drawn from Wikipedia, arXiv, PubMed, and SEC proxy statements.
2. A construction pipeline for page-level supervision at scale: the pipeline combines HTML-grounded page rendering, evidence-page configuration sampling, VLM-based query generation, and a query-cleaning stage that filters for clarity, correctness, closedness, and absence of explicit layout references.
3. A six-way evidence-structure taxonomy: Page-Single, Page-Multi, Doc-Adjacent, Doc-Near, Doc-Far, and Cross-Doc, separating ordinary page lookup from within-document and cross-document distributed evidence.
4. A systematic retrieval evaluation: base retrievers are compared with OCR-based, text-based, and hybrid baselines, plus diagnostic settings including query splitting, document-context scoring, reranking, and round-robin ensembles.

Main Findings

• Multi-vector beats single-vector: comparing Jina V4 single-vector and multi-vector variants, the multi-vector version improves nDCGAll@10 and RecallAll@10 by 11.6 and 10.5 percentage points under Macro (12.9 and 10.0 under Micro), indicating that token-level or patch-level matching matters for recovering complete evidence sets.
• Strongest evaluated configuration: ColNomic (Ensemble) reaches 69.28 nDCGAll@10 and 80.64 RecallAll@10 under structure-balanced Macro, and 75.64 and 86.23 under Micro. The abstract reports this rounded as 69.3 nDCGAll@10 and 80.6 RecallAll@10 under Macro.
• Evidence-complete retrieval is much harder than ordinary page relevance: every evaluated setting loses substantial ground when the requirement becomes retrieving all evidence pages rather than any relevant page.
• Document-context scoring helps most within a document: for ColPali, document-context scoring improves Macro RecallAll@10 from 49.13 to 65.69 (Micro 60.26 to 69.90), while reranking gives the best nDCGAll@10 under both protocols. For ColNomic, combining base and context lists (B+Ctx) gives the best Macro nDCGAll@10 and RecallAll@10 among the settings shown in that table.
• Cross-document retrieval remains the hardest regime: the best Cross-Doc RecallAll@5 reaches 50.9 on Wikipedia, 26.6 on SEC, 18.0 on arXiv, and 28.0 on PubMed. For ColNomic's best variant alone, Cross-Doc RecallAll@5 is reported as 49.8 on Wikipedia, 26.6 on SEC, 14.1 on arXiv, and 20.0 on PubMed.
• The sharpest drop is between within-document and cross-document evidence: in the ColNomic B+Ctx+Rank evidence-structure view, source-macro RecallAll@5 is 93.4 for Page-Single (86,196 queries) and 94.4 for Page-Multi (120,624 queries), falls to 71.4 for Doc-Adjacent (11,972 queries), 69.0 for Doc-Near (3,392 queries) and 66.3 for Doc-Far (2,563 queries), then collapses to 27.6 for Cross-Doc (47,013 queries).
• OCR-BM25 is the strongest text baseline: it achieves the highest Macro RecallAll@10 of 61.69 among the text baselines. Fusing it with ColNomic (Ctx) raises Macro RecallAll@10 from 78.19 to 78.67 while lowering nDCGAll@10 from 66.03 to 65.13.
• Reranking fixes fine-grained confusions: in a qualitative Cross-Doc example, base ColQwen2.5 retrieves only one of the two relevant evidence pages, whereas the reranked version retrieves both; the base model's retrieved pages were thematically plausible but referred to incorrect mission numbers.
• Benchmark quality checks: of 800 manually reviewed entries, 746 were entirely correct, an observed accuracy of 93.25%, with Cross-Doc accuracy at 88.5%. For the 47,013 Cross-Doc queries and their 100,504 evidence elements, stored and bounding-box-derived page assignments matched perfectly for 84.23% of queries.
• Query cleaning removes about 30% of generations: the dataset is reduced from 388,389 generated queries to 271,760 final benchmark queries, validated against a manually annotated query-cleaning evaluation set of 1,438 samples.

Methodology in Plain English

The authors start by assembling a document collection rather than reusing an existing one. For Wikipedia they begin with 78 manually selected seed articles and use an LLM-guided spider to follow links to related articles. For arXiv they seed with 41 recent survey articles, then add 20,000 articles drawn from astrophysics, condensed matter, quantitative biology, and electrical engineering categories, giving 36,132 indexed articles to account for later removal. For PubMed they pull 9,488 open-access articles from 2025 to 2026 along with license information. For SEC they take the most recent proxy statement from about 2,000 large publicly listed companies, using only one statement per company to avoid ambiguity across years.

Documents are then rendered with headless Chrome via Playwright. Before rendering, the pipeline triggers lazy-loaded content, strips invisible elements and source-specific clutter, injects identifiers into HTML elements, and records their bounding boxes. Each full-page screenshot is split into page images of 2000 × 4000 pixels at a rendering scale factor of 2, using whitespace and bounding-box heuristics to avoid cutting tables, figures, or text blocks in half.

Queries come next. The pipeline samples evidence configurations according to the target structure — visually rich top pages for Page-Single and Page-Multi, TF-IDF cosine similarity page pairs with a content-richness term for within-document multi-page types, and embedding-based document pairing followed by the same page-pair strategy for Cross-Doc. A vision-language model receives the rendered pages, the HTML with injected identifiers, and a page analysis, and produces queries, answers, and evidence element identifiers. Invalid outputs, including malformed JSON or bad identifiers, are discarded. A final cleaning stage scores each query on clarity, answer correctness, closedness, and absence of explicit layout references such as "in the table" or "in the figure", using LLM-generated scores and a binary removal classifier.

Evaluation ranks pages from the shared corpus and reports evidence-complete metrics, RecallAll@k and nDCGAll@k, which are zero unless every ground-truth evidence page appears in the top-k list. The primary aggregate is structure-balanced Macro, giving equal weight to the six evidence structures; Micro, the mean over all queries, is secondary. Reported cutoffs are k ∈ {3, 5, 10}.

Why This Matters

Impact on research. The paper argues that long-document visual retrieval is not just a scale problem but a different evaluation regime with different bottlenecks. It provides a diagnostic target that separates candidate generation, document-level coverage, and candidate ordering, and it shows that aggregate scores mask large cross-document failures that source- and structure-level breakdowns expose.

Real-world applications:

• Question answering over scientific literature, where an answer may require a result stated in prose, a table of numbers, and a later figure.
• Financial and regulatory analysis over proxy statements, which average 41.97 pages in this corpus and contain repeated structures and visually plausible distractors.
• Encyclopedic and educational assistants that must gather evidence from several linked articles rather than a single snippet.
• Enterprise document search over long, visually heterogeneous reports with charts, tables, and layouts that plain-text representations lose.

Industry relevance. Systems built around retrieval-augmented generation need the full evidence set before a downstream model can answer correctly. The paper explicitly notes that a high RecallAll@10 means all required pages were retrieved but does not guarantee a downstream task will answer correctly, and that the experiments do not show vision-only retrieval is superior to OCR-only, text-only, or hybrid retrieval. The strongest hybrid result in the paper comes from fusing ColNomic (Ctx) with OCR-BM25 via reciprocal rank fusion, which suggests complementary value across representation types.

Future Directions

• Element-level grounding: the benchmark currently annotates evidence at the page level rather than exhaustively at the element level, which limits how precisely evidence location can be graded.
• Beyond HTML-born documents: the corpus covers rendered HTML-born documents only, leaving scanned and handwritten documents outside scope.
• Reducing generator and filter bias: because queries are VLM-generated and automatically cleaned, generator style and filtering criteria may still shape the benchmark distribution, so alternative generation strategies are an open question.
• Cross-document retrieval methods: the paper reports that query splitting can be more effective than reranking for cross-document regimes, but cross-document evidence coverage remains limited across all evaluated configurations, and the paper does not evaluate end-to-end question answering over retrieved evidence.

Target Audience

Researchers and engineers working on visual document retrieval, multimodal embeddings, and retrieval-augmented generation over long documents; benchmark designers interested in evidence-complete evaluation and structure-stratified analysis; and practitioners building document search over scientific articles, encyclopedic pages, biomedical literature, or financial filings.

Authors’ abstract

Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval. The new dataset and code will be made publicly available at https://yufanchen96.github.io/VL-DocIR.