FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation

AI summary

Overview

Research area: Evaluation of retrieval-augmented generation (RAG) systems, multimodal document understanding, and hallucination detection in large language models.

Technical level: Intermediate. Readers need some familiarity with RAG pipelines, embeddings, OCR, and benchmark design.

Scope: The paper presents FATHOMS-RAG, a benchmark and evaluation framework for assessing whole RAG pipelines on multimodal scientific documents, combining a 93-question curated dataset, a phrase-level recall correctness metric, and a nearest-neighbor classifier that separates abstentions from hallucinations.

What This Paper Is About

RAG is widely used to ground language model outputs in external documents, but existing benchmarks evaluate only parts of a pipeline (retrieval quality, or generation given context) rather than the whole system, and few address documents where key information lives in tables and images. The authors build a small, human-curated benchmark from 8 open-access scientific papers and use it to compare text-only, OCR-enhanced, and closed-source multimodal pipelines on correctness and hallucination behavior. The goal is a lightweight, reproducible way to measure whether a pipeline can ingest, retrieve, and reason across text, tables, and images in one or several documents.

Key Contributions

1. A human-created dataset of 93 questions drawn from 8 open-access (CC-BY, arXiv-hosted) scientific papers, organized into five categories: text-only, tables, images, multimodal, and cross-document multimodal.
2. A phrase-level recall correctness metric that scores a predicted answer as a float between 0.0 and 1.0 by taking the maximum fraction of required phrases matched across all acceptable answer variations.
3. A nearest-neighbor embedding classifier for hallucination detection that labels responses as "statement" or "abstention," and flags a hallucination when an answer is stated as fact but does not reach full phrase-level recall. This avoids reliance on token-level probabilities or hidden states, which are unavailable in closed-source pipelines.
4. A comparative evaluation of two open-source retrieval pipelines (LlamaIndex text-only; Docling with EasyOCR) across multiple local models, and four closed-source multimodal APIs (Claude Sonnet 4, Gemini 2.5 Flash, GPT-4.1, GPT-4o), plus a third-party human evaluation of the automatic metrics.

Main Findings

• Text-only pipelines fail badly outside plain text: With LlamaIndex, text-only correctness reached up to 0.63 (gpt_oss:20b) and 0.62 (llama3_3:70b and gpt_oss:120b), but table-based queries rarely exceeded 0.23 and image-based queries often fell below 0.21. Multimodal correctness stayed at 0.13–0.34, and hallucination rates in those categories regularly surpassed 60–80%, reaching as high as 86%. Overall correctness ranged from 0.19 (gemma3:1b) to 0.32 (llama3_3:70b), with hallucination from 0.44 to 0.75.
• OCR and layout-aware ingestion help, especially on images and cross-document questions: Docling with EasyOCR raised text-only correctness to as high as 0.78 (llama3_3:70b) with hallucination rates down to 0.16. Image questions reached up to 0.36 correctness with hallucination as low as 0.14 for llama3_3:70b. Cross-document multimodal correctness reached 0.55 for gpt_oss:120b and gpt_oss:20b, versus LlamaIndex models rarely exceeding 0.29. Overall correctness ranged from 0.28 (gemma3:1b and gemma3:4b) to 0.40 (gpt_oss:120b), with hallucination spanning 0.39 to 0.61. Table queries remained weak (typically at or below 0.23).
• Closed-source APIs outperform both open-source pipelines: Overall correctness ranged from 0.71 to 0.82 (peaking at 0.82 for Claude Sonnet 4), with hallucination rates between 0.20 and 0.32. On text-only and table queries, all evaluated closed-source models were at or above 0.90 correctness with hallucination at or below 0.10. Image questions reached up to 0.86 correctness with hallucination as low as 0.14.
• Cross-document multimodal reasoning is a universal bottleneck: Even closed-source systems fell to 0.48–0.64 correctness in this category with hallucination rates reaching 0.60, making it the hardest category across every pipeline type.
• Automatic metrics aligned well with human judgment: A third-party reviewer rated agreement on a 1–5 Likert scale. Average agreement was 4.62 for correctness and 4.53 for hallucination detection. By pipeline, Docling with EasyOCR scored 5.00 and 4.89, LlamaIndex RAG 4.44 and 3.94, and closed-source 4.42 and 4.75.
• A known classifier gap: Phrases such as "the provided excerpt does not contain" were not classified as abstentions by the current classifier; these appeared twice in the LlamaIndex answers used in the human study, lowering that category's hallucination agreement. The authors created an updated statement/abstention training set for future evaluations.

Methodology in Plain English

The authors assembled a question set by hand from scientific papers, ensuring each question requires information from a specific modality or combination of modalities (text, a table, an image, several modalities in one paper, or modalities across multiple papers). Each question has one or more acceptable answers, and each acceptable answer is broken into required key phrases. A model's answer gets credit proportional to how many of those required phrases appear, taking the best-scoring acceptable variation.

To separate refusals from false claims, they embed a small set of human-labeled example answers as either "statement" or "abstention" using a sentence embedding model, then classify each prediction by its nearest neighbor under cosine distance. Any response classified as a statement that does not reach full phrase-level recall is counted as a hallucination; abstentions are not counted as hallucinations even if they don't answer the question.

Three pipeline configurations were tested. The LlamaIndex pipeline ingests PDFs with PyMuPDF, splits pages into overlapping chunks using SentenceSplitter (chunk size 1024 characters, 200-character overlap), embeds them with the Ollama embedding model nomic-embed-text, retrieves the top k=3 chunks by cosine similarity, and passes them to the model — text only, with images and tables ignored. The Docling with EasyOCR pipeline converts PDFs to markdown, performs OCR on all pages, and extracts table structures with cell matching enabled; at query time it injects the relevant document content directly into the prompt, truncated to 15,000 characters if needed. The four closed-source APIs (Claude Sonnet 4, Gemini 2.5 Flash, GPT-4.1, GPT-4o) were called through vendor APIs and natively accept multimodal input. All questions come from 8 arXiv papers, and the dataset and code are released publicly.

Why This Matters

This work targets a gap: prior benchmarks such as MMMU, NPHardEval4V, CRAG, and RAGAS evaluate pieces of the problem (multimodal reasoning, generation under retrieval, or individual RAG components), but none assesses a full pipeline from ingestion through retrieval, reasoning, and generation on multimodal documents. The framework also offers a practical way to detect hallucinations without access to model internals — a requirement that rules out semantic entropy and hidden-state methods when closed-source pipelines are being compared.

Real-world applications:

• Scientific and technical document assistants: Researchers querying large corpora of PDFs where critical values sit in tables and figures rather than prose.
• Enterprise knowledge systems: Organizations whose internal reports, scans, and spreadsheets mix text, tables, and images, and where confident wrong answers carry real cost.
• Regulated and high-stakes domains: Settings such as medicine, law, or engineering where distinguishing "I can't answer from this context" from a fabricated answer is essential.
• Procurement and model selection: Teams comparing open-source versus proprietary RAG stacks need category-level evidence of where each option breaks down.

Industry relevance: The results quantify a persistent gap between open-source and closed-source pipelines, and show that OCR and layout-aware preprocessing narrow but do not eliminate it. They also indicate that no current system, proprietary included, handles cross-document multimodal reasoning reliably — a direct constraint on deploying RAG for document-heavy professional workflows. The released dataset enables fast comparison (roughly one hour for most open-source pipelines, as the authors report).

Future Directions

• Expand the dataset substantially in size, number of documents, and subject-area coverage; the authors specifically propose datasets built around massively cross-document questions that supply 100 or more relevant chunks from various documents to test reasoning as well as retrieval.
• Distinguish types of abstention, separating epistemic abstentions (insufficient information) from alignment-based abstentions (refusals due to safety or ethical concerns).
• Improve the hallucination classifier by folding human-evaluation feedback and a broader set of annotated statement/abstention examples into the training data.
• Study tool use outside a traditional RAG pipeline, for example whether calling an external tool to generate a map of past dig sites improves quality for a researcher-facing system.
• Address comparison limits between open-source and closed-source systems, since visibility into closed-source retrieval mechanisms is limited and features such as web search may compensate for weaker retrieval.

Target Audience

Researchers and practitioners building or evaluating RAG systems over multimodal documents; benchmark and evaluation specialists interested in correctness and hallucination metrics; engineers choosing between open-source and proprietary retrieval stacks; and teams working with scientific, technical, or scanned PDF collections where tables and figures carry essential information. The paper is written at an intermediate level and assumes familiarity with retrieval, embeddings, and OCR concepts.

Authors’ abstract

Retrieval-augmented generation (RAG) has emerged as a promising paradigm for improving factual accuracy in large language models (LLMs). We introduce a benchmark designed to evaluate RAG pipelines as a whole, evaluating a pipeline's ability to ingest, retrieve, and reason about several modalities of information, differentiating it from existing benchmarks that focus on particular aspects such as retrieval. We present (1) a small, human-created dataset of 93 questions designed to evaluate a pipeline's ability to ingest textual data, tables, images, and data spread across these modalities in one or more documents; (2) a phrase-level recall metric for correctness; (3) a nearest-neighbor embedding classifier to identify potential pipeline hallucinations; (4) a comparative evaluation of 2 pipelines built with open-source retrieval mechanisms and 4 closed-source foundation models; and (5) a third-party human evaluation of the alignment of our correctness and hallucination metrics. We find that closed-source pipelines significantly outperform open-source pipelines in both correctness and hallucination metrics, with wider performance gaps in questions relying on multimodal and cross-document information. Human evaluation of our metrics showed average agreement of 4.62 for correctness and 4.53 for hallucination detection on a 1-5 Likert scale (5 indicating "strongly agree").