MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation

AI summary

Overview

Research area: Evaluation benchmarks and dataset generation for Retrieval-Augmented Generation (RAG), combining multi-agent LLM orchestration, multimodal document understanding, and multi-hop question answering.

Technical level: Advanced. The paper includes formal probabilistic framing (joint probability of valid QA pair generation), optimization objectives for semantic chunking, and clustering-based deduplication formulas, alongside empirical benchmark tables.

Scope: The paper introduces MiRAGE, a multi-agent framework that automatically generates verified, domain-specific, multimodal, multi-hop question-answer datasets from proprietary technical corpora so that multimodal RAG systems can be rigorously benchmarked.

What This Paper Is About

Existing RAG evaluation benchmarks (such as Natural Questions and MS MARCO) come from open-domain text corpora, so they do not reflect the realities of high-stakes enterprise documents where knowledge is scattered across text, tables, charts, and diagrams. The authors argue that current synthetic dataset generators use linear pipelines without feedback mechanisms, which produces hallucinated or semantically redundant evaluation data. MiRAGE addresses this by orchestrating a swarm of specialized agents that recursively gather disjoint evidence, inject an expert persona into question generation, and adversarially verify every generated answer against its source context.

Key Contributions

1. A model-agnostic multi-agent framework for multimodal ingestion. MiRAGE automates ingestion and semantic segmentation of complex multimodal documents while preserving the semantic dependencies between text and visual elements, using a vision-language description agent and a semantic chunking agent.

2. A generative methodology based on recursive context expansion and persona injection. Questions are built from a recursively expanded semantic context rather than a single seed chunk, and are conditioned on an inferred domain and expert persona, producing questions that require logical deduction across disjoint chunks rather than extractive lookup.

3. An empirical evaluation across four domains. The authors report results for regulations, finance, science, and journalism, showing higher reasoning complexity (average hop count above 2.3 in the technical domains) and adherence to the latent thematic distribution of the source corpus, measured with Jensen-Shannon divergence.

4. A component-wise ablation study. The ablation on a subset of the S&P Global annual reports isolates the contributions of multihop context, the QA verifier agent, domain/persona injection, chunking strategy, and multimodal representation choices.

Main Findings

• Reasoning complexity is consistently multi-hop in technical domains. For finance, regulation, and science corpora, the average hop count H exceeds 2.3, peaking at 2.84 with Gemini 2.5 Flash on the S&P Global dataset (GPT 5 Mini: 2.42 on the same corpus). UNECE GTRs reached 2.45 (Gemini 2.5 Flash) and 2.60 (GPT 5 Mini); Q-Bio Arxiv reached 2.35 (Gemini 2.5 Flash) and 2.55 (GPT 5 Mini).

• Journalism content produces much shallower questions. The NYTimes corpus averaged H ≈ 1.2 (1.10 for Gemini 2.5 Flash, 1.25 for GPT 5 Mini). The paper attributes this to the open, less connected nature of journalistic content and states that the corpus did not contain chunks relevant to the queries needed to make the context complete.

• Faithfulness and relevance are high. Faithfulness scores were consistently above 0.91 for three of the four domains, and relevance exceeded 0.81 across all experiments. Reported faithfulness values range from 0.81 (GPT 5 Mini, Q-Bio Arxiv) to 0.96 (Gemini 2.5 Flash on S&P Global and GPT 5 Mini on UNECE GTRs). The authors credit the adversarial verifier agent for filtering hallucinations.

• Visual grounding remains weak. The highest visual grounding score reported in the main results is 0.45 (GPT 5 Mini on UNECE GTRs); scores reach only 0.21 for Gemini 2.5 Flash on S&P Global. Of the 1093 QA pairs generated for the finance domain, only 84 are multimodal, indicating that the VLMs prefer textual content. The authors hypothesize that the generated image descriptions made visual elements partially redundant.

• Removing the multihop context collapses difficulty. In the ablation, difficulty dropped from 0.85 to 0.61, and the domain-alignment divergence jumped to JSD ≈ 4.35 versus 0.08 for the baseline, indicating reversion to simple extractive QA.

• Removing the verifier agent causes the largest faithfulness failure. Faithfulness fell from 0.97 to 0.74 and relevance from 0.95 to 0.76. The authors note qualitatively that the generation agent frequently hallucinated relationships between unconnected data points to satisfy complexity requirements.

• Domain and persona injection drives question depth. Ablating domain analysis dropped difficulty from 0.85 to 0.52, with the model defaulting to generic rather than expert-level questions.

• Semantic chunking results are explicitly described as inconclusive on hop count. Replacing agentic semantic chunking with a fixed 2048-token window raised average hops only marginally (1.92 to 2.01), which the authors say makes it inconclusive to highlight the importance of semantic chunking for hop count; they point instead to drops in faithfulness and relevance.

• Image-only configurations trade grounding for faithfulness. The image-only setting produced higher visual grounding (0.62) but the lowest faithfulness (0.71), while the description-only setting performed comparably to full MiRAGE. The authors conclude that MiRAGE can be powered by LLMs when textual descriptions of images are available.

• No direct head-to-head comparison with prior generators is reported. The paper discusses DataMorgana, DQABench, SMMQG, RAGProbe, and WeQA qualitatively in the literature review, but the results tables compare MiRAGE configurations and two backbone models, not MiRAGE against those frameworks.

Methodology in Plain English

MiRAGE takes a collection of technical documents and runs five phases:

1. Ingestion and chunking. A document layout analysis engine splits the document structurally. For every visual element (figure, table, chart), a description agent powered by a vision-language model writes a dense technical description. Text is then split into semantically coherent chunks using a sliding window; the chunking agent chooses the partition that minimizes dissimilarity between adjacent chunks plus a penalty term that discourages over-fragmentation.

2. Domain and persona recognition. The corpus chunks are embedded, projected into a lower-dimensional space via manifold learning, and clustered into K thematic clusters. Class-based TF-IDF with Maximal Marginal Relevance produces representative keywords per cluster, and an agent synthesizes a domain and an expert role title from them (for example, "Financial Reporting Analyst" for corporate financial reporting).

3. Multihop context building. Starting from a seed chunk, a completeness-checking agent decides whether the context is self-contained. If not, it emits targeted search queries; a hybrid retriever pulls top-N candidates that are reranked, and a separate verification agent decides whether each candidate is explanatory, related, or unrelated before it is added. The loop repeats until the context is judged complete or a maximum iteration count is reached.

4. QA generation and verification. A generation agent conditioned on the assembled context, the domain, and the persona constructs question-answer pairs, explicitly mapping which parts of the question and answer came from which chunk. A separate verifier agent then checks each candidate for factual support by the context and whether the question actually requires that context, discarding failures.

5. Refinement and deduplication. Validated pairs are clustered by question embedding, then answers are sub-clustered to find redundancy. Similarity combines answer-embedding cosine similarity with Jaccard overlap of the two source contexts. When all pairs within a sub-cluster are sufficiently similar, a refinement agent merges them into a single representative pair; otherwise the originals are kept. The final dataset is the union across clusters.

Models and data: The framework is model-agnostic; the authors primarily used Gemini-2.5-Flash and GPT-5-Mini for the reasoning agents, the Nomic model for multimodal embeddings, and an LLM-as-a-reranker for retrieval precision. They note that CLIP- and SigLIP-based embedding and reranker models have token limits that make them ineffective for these multimodal tasks. Corpora are the S&P Global Annual Reports (finance), UNECE Global Technical Regulations (regulation), arXiv quantitative biology submissions from January 2025 (science), and New York Times Opinions from the Visual News dataset (journalism). Metrics include faithfulness, relevance, average hop count, VLM-verified visual grounding, an expert-rated difficulty score on a 0-10 scale normalized to 0-1, and Jensen-Shannon divergence between the topic distributions of the corpus and the generated dataset.

Why This Matters

Impact on research. The paper targets a specific gap: benchmarks that simultaneously cover domain specificity, multimodality, and multi-hop reasoning. By automating gold-standard dataset creation that mirrors the latent thematic structure of a private corpus, it provides a way to stress-test RAG systems without relying on open-domain datasets or expensive human annotation.

Real-world applications:

• Enterprise technical documentation search — technical standards, manuals, and regulations where definitions, tables, and conditional logic are spread across thousands of pages (UNECE GTRs here span 7594 pages and 3.8M tokens).
• Financial analysis and reporting — annual reports where strategic narrative is interwoven with dense tables and financial charts (S&P Global: 1302 pages, 1,120 images, 2,800 tables).
• Scientific literature retrieval — quantitative biology papers with extreme lexical specificity and complex visual artifacts such as 3D molecular renderings (8336 pages, 9,400 images).
• Media and journalism question answering — opinion content, though the paper shows this domain yields notably shallower multi-hop structure (average hops around 1.2).

Industry relevance. The work is authored at ABB Inc and framed around high-stakes enterprise deployments, where hallucination in RAG outputs is costly. It gives organizations a method to build evaluation sets from their own proprietary corpora, and it shows that the framework can be driven by LLMs alone if image descriptions are available — a practical cost consideration for teams without strong vision model access.

Future Directions

• Reducing computational cost. The authors state explicitly in their Limitations section that the multi-agent architecture is computationally intensive, with the multihop context building and QA verification loops driving higher token costs and latency, and that future work will optimize the agentic workflow for token efficiency.
• Democratizing the framework with open-source models. The limitations section names exploring open-source model performance as a goal; the current results only cover Gemini 2.5 Flash and GPT 5 Mini.
• Improving visual grounding. The paper calls visual grounding a frontier that "still remains," with maximum reported scores of 0.45 in the main results; a plausible open question is whether the low multimodal QA share (84 of 1093 finance pairs) can be raised and whether description-based redundancy can be avoided.
• Establishing the value of semantic chunking. The authors themselves flag the chunking ablation as inconclusive on hop count, leaving open whether a more sensitive measurement can demonstrate the benefit they suspect from the faithfulness and relevance drops.
• Comparison against other synthetic generators. No head-to-head quantitative comparison with frameworks such as DataMorgana, DQABench, SMMQG, RAGProbe, or WeQA is reported, so relative quality remains an open question.

Target Audience

This paper benefits RAG practitioners and evaluation engineers who need domain-specific benchmarks for proprietary document collections; enterprise AI teams in regulated or technical industries (finance, standards, engineering, science); researchers working on synthetic dataset generation, agentic pipelines, and multimodal retrieval; and dataset and benchmark builders interested in multi-hop question generation and adversarial answer verification. Readers should be comfortable with retrieval metrics, embedding-based clustering, and multi-agent LLM architectures.

Authors’ abstract

The rapid evolution of Retrieval-Augmented Generation (RAG) toward multimodal, high-stakes enterprise applications has outpaced the development of domain specific evaluation benchmarks. Existing datasets often rely on general-domain corpora or purely textual retrieval, failing to capture the complexity of specialized technical documents where information is inextricably multimodal and reasoning requires synthesizing disjoint evidence. We address this gap by introducing MiRAGE, a Multiagent framework for RAG systems Evaluation, that leverages a collaborative swarm of specialized agents to generate verified, domain-specific, multimodal, and multi-hop Question-Answer datasets. MiRAGE orchestrates a swarm of specialized agents: a recursive context optimization loop to aggregate scattered evidence, an adversarial verifier agent to guarantee factual grounding, and an agent to recognize the expert persona and the relevant domain to mimic expert cognitive workflows. Extensive empirical evaluation across four distinct domains (regulations, finance, quantitative biology, and journalism) demonstrates that MiRAGE generates datasets with significantly higher reasoning complexity (>2.3 average hops) and factual faithfulness. Our ablation studies point that MiRAGE can be powered by LLMs if textual descriptions of the images are available. Visual grounding still remains a frontier. By automating the creation of gold standard evaluation datasets that reflect the latent thematic structure of proprietary corpora, MiRAGE provides the necessary infrastructure to rigorously benchmark the next generation information retrieval systems.