MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation

AI summary

Overview

Research area: Retrieval-augmented generation (RAG), multimodal large language models (MLLMs), and knowledge-graph-based question answering over long, visually rich documents.

Technical level: Advanced. The paper assumes familiarity with RAG pipelines, knowledge graphs, dense vector retrieval, and LLM-as-judge evaluation protocols.

Scope: The paper introduces MegaRAG, a system that automatically builds multimodal knowledge graphs from text, figures, tables, and page images, then uses them to answer both book-level ("global") and page- or slide-level ("local") questions. Code is released at https://github.com/AI-Application-and-Integration-Lab/MegaRAG (arXiv:2512.20626v2 [cs.AI]).

What This Paper Is About

RAG lets large language models pull in external documents at query time, but context-window limits make it hard for these models to reason across entire books or long reports, especially when pages mix prose with charts, diagrams, and layouts. Existing knowledge-graph RAG methods (GraphRAG, LightRAG) help by structuring text into entity-relation graphs, but they are text-only: they ignore visual cues, and because they extract entities chunk-by-chunk, their graphs miss relationships that cross chunk or page boundaries. MegaRAG's goal is to build the knowledge graph itself multimodally — incorporating text, figures, tables, and whole-page images — and to use that graph in both retrieval and answer generation.

Key Contributions

1. MegaRAG, an automatic multimodal knowledge graph (MMKG) builder for visual document QA. The system uses MLLMs to extract page-level entities and relations in parallel across all pages of a document, then merges them into a unified graph, requiring no fine-tuning and no manually curated domain graphs.

2. A page-based, two-round refinement process. A first pass produces an initial graph; a second pass gives each page a retrieved subgraph from that initial graph as global context so the model can recover cross-modal links (for example, a chart that illustrates a claim in the surrounding text) that isolated per-chunk extraction loses. The subgraph approach keeps inputs lightweight enough for the MLLM's context window.

3. A retrieval and generation design that handles text and vision together, including dual-level keyword retrieval over the graph, supplementary page-image retrieval, and a two-stage generation scheme that first answers separately from the graph and from page images and then fuses the two answers to reduce modality bias.

4. Evaluation across global and local QA on both textual and multimodal benchmarks, where the authors report consistent wins over NaiveRAG, GraphRAG, LightRAG, and multimodal RAG baselines (VisRAG, GME, ColQwen), plus ablations and a computational-cost analysis.

Main Findings

• Textual global QA (UltraDomain): MegaRAG reports average win rates of 59.0% for Comprehensiveness, 71.4% for Diversity, 74.8% for Empowerment, and 71.8% Overall across the Agriculture, Computer Science, Legal, and Mixed-Domain subsets. The authors attribute this partly to refining a global knowledge graph rather than relying on per-page gleaning as GraphRAG and LightRAG do.

• Multimodal global QA: On World History, Environmental Report, DLCV, and GenAI, MegaRAG reports average win rates of 83.3% (Comprehensiveness), 92.7% (Diversity), 84.7% (Empowerment), and 89.5% (Overall). Against GraphRAG specifically, Overall win rates are reported at 92.8% (DLCV), 93.6% (World History), 84.8% (Environmental Report), and 99.2% (GenAI). Gains are described as largest on slide-based datasets where core content is visual rather than textual.

• Multimodal local QA: MegaRAG reaches 64.85% accuracy on SlideVQA (2k), which the paper states is more than double the strongest baseline (LightRAG at 27.66%). On RealMMBench it reports 39.51% (FinReport), 58.37% (FinSlides), 51.51% (TechReport), and 60.86% (TechSlides), surpassing LightRAG by 8 to 9 percentage points on the more text-heavy FinReport and TechReport subsets and by roughly 45 and 29 percentage points on FinSlides and TechSlides.

• Against multimodal RAG baselines: In pairwise LLM-judge comparisons with VisRAG, GME, and ColQwen on DLCV, World History, Environmental Report, and GenAI, MegaRAG wins on all datasets and all four evaluation dimensions — for example, an Overall win rate of 78.4% against GME on World History and 86.4% against GME on GenAI.

• Ablation — MMKG retrieval is the most critical piece: Removing MMKG retrieval (page retrieval only, variant A2) causes the largest drop; on DLCV it records 0.0% Overall wins against full MegaRAG, and on World History 0.8%. Text-only graph construction (A1) degrades performance consistently, especially on visually intensive documents. Replacing two-stage generation with single-pass generation (A3) causes moderate, consistent regressions, largest in Diversity and Empowerment.

• Cost: Build time and KG token usage are roughly 1.4× those of text-only GraphRAG because MegaRAG processes 17M to 25M visual tokens. On World History (788 pages), GraphRAG takes 23.0 minutes and 1.2M KG tokens total, while MegaRAG takes 19.0 minutes for the initial stage plus 12.0 minutes for refinement (31.0 minutes total) and 38.2M KG tokens. On the Environmental Report (422 pages), GraphRAG takes 10.0 minutes and 0.6M tokens, while MegaRAG takes 8.5 + 5.5 minutes (14.0 minutes total) and 54.3M tokens. Graph construction is a one-time offline process that can be reused across queries, and it lets MegaRAG capture 473 to 538 figures that text-only methods cannot.

• Inference latency: On the Environmental Report with a single RTX 3090 (24GB), retrieval takes about 1.0s (MMKG and page retrieval run concurrently), Stage 1 generation takes about 20.5s from the subgraph and 26s from page images in parallel, and Stage 2 fusion takes about 16s, for roughly 42s total per question. GME needs about 26.4s (0.4s retrieval plus 26s generation), so MegaRAG adds about 15.6s.

Methodology in Plain English

MegaRAG treats each document page as a bundle of four things: its text, its figure images, its table images, and a full-page rendered image that captures layout. In the initial stage, an MLLM (GPT-4o-mini in the implementation) reads every page independently and in parallel, pulling out entities and relations. Text produces multiple entities; each figure or table becomes a single entity; decorative visuals like logos or background patterns are ignored; the full-page image is used only for spatial reasoning and produces no nodes. Every entity has a name, a type, and a description; every relation has a source, a target, a description, and keywords. These page-level outputs are then merged into one graph, combining nodes with the same name and edges with matching endpoints and relation types, and pooling their descriptions and keywords.

Because that first pass often misses cross-modal links, a refinement stage follows. For each page, the system retrieves a small, page-specific subgraph from the initial graph — using the entity names and relation keywords already extracted from that page to match against dense embeddings of the graph's entities and relations, then expanding with one-hop neighbors. The same MLLM is then re-prompted with the page plus that subgraph, and it looks for entities mentioned in the page but absent from the subgraph and for implicit relations the page suggests but the graph lacks. The example the paper gives: text saying electric vehicle sales rose in 2023 and a nearby bar chart titled "Annual Sales by Vehicle Type" are initially separate entities; refinement adds a relation such as illustrates or supports between them. Only one refinement round is used, though the process is described as iterative in principle.

For indexing, page images are encoded directly with the multimodal encoder GME (GME-Qwen2-VL-2B), entity embeddings are built from name plus description, and relation embeddings from keywords, endpoint names, and description, with all vectors stored in separate dense stores by type. At query time, the MLLM extracts low-level keywords (specific entities) and high-level keywords (broader concepts); both are embedded and combined to retrieve the top-k entities and top-k relations (k = 60), and retrieved entities are expanded with one-hop neighbors. In parallel, a text-to-image search retrieves the top-m pages (m = 6).

Generation is deliberately split into two stages. Because mixing graph text and page images in one prompt tends to make the model over-focus on one modality, the MLLM first produces two intermediate answers independently — one from the retrieved subgraph, one from the retrieved page images — and then a second stage fuses them into the final answer.

Evaluation follows GraphRAG/LightRAG conventions. Global questions are generated by giving an LLM a document outline and having it create five synthetic RAG users, each with five information-seeking tasks and five questions per task, yielding 125 questions per dataset; since these have no gold answers, responses are scored by pairwise LLM judge on Comprehensiveness, Diversity, Empowerment, and Overall

Authors’ abstract

Retrieval-augmented generation (RAG) enables large language models (LLMs) to dynamically access external information, which is powerful for answering questions over previously unseen documents. Nonetheless, they struggle with high-level conceptual understanding and holistic comprehension due to limited context windows, which constrain their ability to perform deep reasoning over long-form, domain-specific content such as full-length books. To solve this problem, knowledge graphs (KGs) have been leveraged to provide entity-centric structure and hierarchical summaries, offering more structured support for reasoning. However, existing KG-based RAG solutions remain restricted to text-only inputs and fail to leverage the complementary insights provided by other modalities such as vision. On the other hand, reasoning from visual documents requires textual, visual, and spatial cues into structured, hierarchical concepts. To address this issue, we introduce a multimodal knowledge graph-based RAG that enables cross-modal reasoning for better content understanding. Our method incorporates visual cues into the construction of knowledge graphs, the retrieval phase, and the answer generation process. Experimental results across both global and fine-grained question answering tasks show that our approach consistently outperforms existing RAG-based approaches on both textual and multimodal corpora.