TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning

AI summary

Overview

• Research area: Multimodal retrieval-augmented generation (RAG) and GraphRAG evaluation, specifically document question answering over visually rich, layout-heavy documents.
• Technical level: Advanced. The paper assumes familiarity with GraphRAG architectures (LightRAG, HippoRAG, Microsoft GraphRAG), multimodal retrieval systems, and RAG evaluation metrics.
• Scope: The paper introduces TopoGraphRAG-Bench, a 2,024-question, 201-document benchmark that tests whether retrieval systems can recover the intended evidence topology (single-hop, bridge-chain, or multi-source synthesis) across text, figures, and tables in complex page layouts.

What This Paper Is About

Real documents spread the evidence needed to answer a question across text passages, tables, figures, and captions scattered over complex page layouts, so a system must do more than find a relevant passage—it must recover the dependency structure connecting heterogeneous evidence units. Existing GraphRAG benchmarks are largely text-centered, while multimodal document RAG benchmarks measure cross-modal retrieval and generation without directly testing whether a system recovers the intended evidence topology. This paper builds a layout-grounded benchmark to diagnose exactly that gap, then uses it to compare text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems.

Key Contributions

1. Topology-aware multimodal GraphRAG benchmark. The authors introduce TopoGraphRAG-Bench, a layout-grounded benchmark with 2,024 QA pairs over 201 long documents, covering single-hop retrieval, bridge-chain reasoning, and multi-source synthesis across text, figures, and tables.
2. Bottom-up evidence-topology construction. Questions are built from verifiable layout-grounded evidence units rather than whole-document prompts, with counterfactual validation enforcing shortcut resistance, modality necessity, and evidence necessity.
3. Diagnostic evaluation of GraphRAG paradigms. The paper evaluates text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems, showing that current systems still struggle with cross-modal alignment and fine-grained evidence-topology recovery.
4. Quality assurance and annotation. Automatic filtering is followed by a stratified manual audit of 303 questions (15.0% of the benchmark) by two PhD-level reviewers, who reached substantial agreement (Cohen's κ = 0.84).

Main Findings

• Multimodal GraphRAG leads overall. On the aggregate results across all 2,024 questions, MegaRAG obtains the highest point estimates for answer accuracy (0.5350), faithfulness (0.9930), response relevancy (0.8172), and step coverage (0.7329), while RAG-Anything achieves the best retrieval scores (context precision 0.8093, context recall 0.7215) and comparable answer accuracy (0.5300).
• Retrieval ranking is not the same as generation ranking. RAG-Anything has the highest context precision and recall, but MegaRAG is stronger on faithfulness, response relevancy, and step coverage, suggesting that layout-grounded reasoning depends on aligning and composing retrieved evidence, not just retrieving it.
• Text-only GraphRAG collapses on multimodal composition. HippoRAG reaches 0.90 accuracy on single-hop text questions but drops to 0.17 on Bridge-MM (aggregated mixed-modality and visual-only bridge questions); LightRAG drops from 0.71 to 0.30 over the same comparison. LightRAG falls to 12.0% accuracy on figure→table paths and 7.5% on table→figure paths, while HippoRAG and Microsoft GraphRAG remain below 10% on both.
• Page-level visual retrieval is not enough. VisRAG performs competitively on single-hop visual questions (0.65 on figure questions, 0.56 on table questions) but falls to 0.26 on Bridge-MM and 0.25 on Synthesis-MM. On bridge paths it reaches 32.9% on figure→figure and 26.6% on table→table, but only 9.3% on figure→table and 18.3% on table→figure.
• Multimodal GraphRAG is more stable but still imperfect. RAG-Anything reaches 0.48 on Bridge-MM and 0.43 on Synthesis-MM; MegaRAG reaches 0.53 and 0.43. Both RAG-Anything and MegaRAG reach 51.8% accuracy on figure→table paths, and MegaRAG reaches 65.0% on table→figure.
• Step coverage tracks the same gaps. On text→text bridge paths, LightRAG obtains 84.6% step coverage but drops to 44.9% on visual-only paths; HippoRAG declines from 65.4% to 23.9%. MegaRAG achieves 83.0% step coverage on visual-only bridge paths, versus 44.9% for LightRAG and 49.4% for VisRAG.
• Synthesis gets harder as evidence becomes heterogeneous. LightRAG reaches 46.6% accuracy on text-only synthesis but drops to 27.4% on cross-modal synthesis; HippoRAG drops from 44.4% to 18.4%. RAG-Anything reaches 43.2% on cross-modal synthesis, and MegaRAG reaches 38.4% on table-only and 42.7% on cross-modal synthesis, but both still trail their own text-only synthesis performance.
• Cost profiles differ sharply. RAG-Anything has the highest normalized indexing time (5,844.99 s/doc) and prompt token usage (2,470,006.99 tokens/doc) among the evaluated systems, while VisRAG is cheapest to index (2.94 s/doc) because it builds no entity-relation graph.

Methodology in Plain English

The authors started from MMDocIR, a multimodal document retrieval corpus, and randomly sampled 201 long, visually rich documents covering nine of MMDocIR's ten application domains. Each layout-level "quote" in those documents was reinterpreted as an evidence unit: text units keep their original text, while figures and tables keep their original image and are augmented with OCR and VLM descriptions. Every unit retains its document ID, page ID, layout ID, modality label, and a semantic representation.

Rather than prompting a large language model over a whole document to generate hard questions, the team built questions bottom-up. They extracted salient entities and concepts from each evidence unit with an LLM, normalized and fuzzy-matched entity mentions, and used recurring entities as semantic anchors linking units. Questions were then generated under three controlled topologies: single-hop questions from an isolated unit, bridge-chain questions composed from two atomic QA pairs linked by a shared entity (with the target question rewritten to replace the explicit entity name with the bridge question's indirect description), and multi-source synthesis questions built from entities that occur in at least five evidence units, retaining only those with at least three distinct evidence facts.

To stop questions from degenerating into easier retrieval cases, the pipeline decouples question generation, answer generation, and quality assessment into separate LLM calls and applies counterfactual checks. For bridge-chain candidates, the pipeline verifies the bridge entity is not revealed in the composed question, that the answer differs from the bridge entity, and that the two hops form a coherent dependency; single-source counterfactuals remove instances answerable from either hop alone. Cross-modal questions are retested with text-only context to remove cases answerable without figures or tables, and synthesis questions undergo leave-one-out ablations over supporting evidence points, keeping only instances with at least three required evidence points. A stratified manual audit of 303 questions (15.0%) followed, with disagreements resolved by discussion and invalid instances revised or removed.

Evaluation compares six systems under fixed model backends: text-only GraphRAG (LightRAG, HippoRAG, Microsoft GraphRAG), page-level visual retrieval (VisRAG), and multimodal GraphRAG (RAG-Anything, MegaRAG). Retrieval is scored with RAGAS context precision and context recall using topology-enriched references that include bridge entities and reasoning traces for bridge chains and required evidence points for synthesis. Generation is scored on answer accuracy, faithfulness, response relevancy, and step coverage, with Qwen3.5-35B-A3B as the judge.

Why This Matters

Impact on research. The paper reframes complex document retrieval as a structure-recovery problem rather than flat relevance ranking, and provides a diagnostic benchmark that separates evidence retrieval from evidence composition. It shows that strong text-derived graph retrieval does not transfer to cross-modal dependencies, and that access to page images does not substitute for fine-grained layout-unit structure—pointing future GraphRAG work toward explicit modeling of layouts, cross-modal alignment, and the reasoning roles of evidence units.

Real-world applications.

• Question answering over financial filings, scientific papers, and technical manuals, where answers depend on combining a chart with a table footnote on another page.
• Enterprise document assistants that must trace a claim across a multi-page report and show which figure, table, and paragraph support it.
• Regulatory and compliance review, where evidence must be assembled from heterogeneous units and the reasoning chain matters as much as the conclusion.
• Medical or clinical literature triage, where a finding in one figure determines which later table becomes relevant.

Industry relevance. The evaluation includes efficiency statistics per document—indexing time, retrieval and generation time, LLM calls, prompt and completion tokens, and graph sizes—which gives practitioners a cost-versus-quality picture. RAG-Anything's high accuracy comes with the heaviest indexing and token footprint (5,844.99 s/doc indexing, 2,470,006.99 prompt tokens/doc), while VisRAG is cheap to index (2.94 s/doc) but limited in compositional reasoning; MegaRAG builds a smaller multimodal graph but shows higher retrieval cost (335.84 s/doc). These tradeoffs matter for anyone deploying document RAG at scale.

Future Directions

• Closing the modality-alignment gap. The remaining variation across ordered bridge paths (for example, 12.0% for LightRAG on figure→table versus 57.6% on text→text) indicates that aligning evidence across modalities remains a central bottleneck for bridge-chain reasoning.
• Better multi-unit composition. Both multimodal GraphRAG systems still trail their own text-only synthesis accuracy, so heterogeneous evidence aggregation is an open problem rather than a solved one.
• GraphRAG beyond text-derived entity graphs. The authors argue for systems that model layout structure, cross-modal evidence alignment, and the reasoning roles of evidence units, not just textual entities and relations.
• Benchmark scope and cost. The paper's own appendix lists limitations of benchmark scope, topology coverage, evaluated systems, and indexing cost as open concerns for follow-up work.

Target Audience

This paper is most useful to researchers and engineers working on retrieval-augmented generation, GraphRAG architectures, and multimodal document understanding—particularly those building or evaluating systems over long, visually rich documents. It also serves benchmark designers and practitioners who need to choose among text-derived graphs, page-level visual retrieval, and multimodal graph systems, and who care about the quality-versus-cost profile of each. Readers without prior exposure to GraphRAG terminology will find the topology framing intuitive, but the evaluation details and baseline comparisons assume an intermediate-to-advanced background.

Authors’ abstract

Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such documents therefore requires more than retrieving relevant passages: systems must recover the evidence topology that connects heterogeneous evidence units. Existing GraphRAG evaluations remain largely text-centered, while multimodal document RAG benchmarks assess cross-modal retrieval and generation without directly evaluating recovery of the intended evidence topology. We introduce TOPOGRAPHRAG-BENCH, a layout-grounded benchmark for multimodal evidence reasoning in GraphRAG, comprising 2,024 questions over 201 long, visually rich documents. Questions are constructed bottom-up from text, figure, and table evidence units under three controlled topologies: single-hop retrieval, bridge-chain reasoning, and multi-source synthesis. To ensure that questions preserve their intended structure, we apply counterfactual validation for shortcut resistance, modality necessity, and evidence necessity. We evaluate text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems using retrieval, generation, and topology-aware reasoning metrics. Multimodal GraphRAG systems achieve the strongest overall performance, but still fail when visual-textual evidence alignment or multi-unit composition is incomplete. Text-only GraphRAG struggles when key dependencies are grounded in figures or tables, while page-level visual retrieval lacks the fine-grained structure needed for topology recovery. These findings motivate GraphRAG systems that move beyond text-derived entity relation graphs to explicitly model document layouts, cross-modal evidence alignment, and the reasoning roles of evidence units. Code and data are available at https://richardlrc.github.io/TopoGraphRAG-Bench/.