Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation

AI summary

Overview

Research area: Natural Language Processing — retrieval-augmented generation (RAG) evaluation, specifically extending factuality and citation metrics to multimodal (text, image, audio, video) sources.

Technical level: Intermediate. The paper assumes familiarity with RAG, factuality metrics like FActScore, and concepts such as subclaim decomposition and entailment-based scoring, but the core ideas are explained with usable background.

Scope: The paper proposes and validates MiRAGE, a claim-centric evaluation framework with two metrics (InfoF1 and CiteF1) for multimodal RAG, comparing it against multimodal adaptations of three prominent text-only RAG metrics across three datasets.

What This Paper Is About

RAG systems increasingly need to generate text grounded in non-text sources like video, audio, and images, yet the metrics used to evaluate them were built for text passages and cannot verify whether claims are actually supported by a video or audio clip. The authors introduce MiRAGE, an evaluation framework that decomposes generated output into atomic "subclaims" and verifies each one—and its citations—against sources in any modality. The goal is a single evaluation methodology that works consistently across text and audiovisual retrieval settings.

Key Contributions

1. MiRAGE framework. A subclaim-centric evaluation approach for multimodal, multi-source, citation-attributed RAG, composed of InfoF1 (factuality precision and information coverage recall) and CiteF1 (citation support precision and proper attribution recall). Both work in reference-based and reference-free (collection) settings.

2. Multimodal adaptations of existing metrics. The authors implement multimodal versions of three prominent text-only RAG metrics—ALCE, ARGUE, and RAGAs—and document where each breaks down on audiovisual sources.

3. Human alignment study. They collect three types of human judgments (extrinsic quality, intrinsic claim, and grounding judgments) on audiovisual RAG outputs, and show that human-computed MiRAGE scores align strongly with holistic quality judgments, validating the framework's design independent of any automatic verifier.

4. Empirical comparison across modalities. Evaluation on three datasets spanning multilingual text report generation (RAGTIME) and audiovisual article generation (WikiVideo, WikiVideo-25), plus a claim-grounding benchmark across four modality settings.

Main Findings

• MiRAGE best matches human judgments, on text and beyond. On RAGTIME, MiRAGE achieved the highest agreement with human annotations on both information coverage (Kendall's τ = 77.7) and citation support (79.2), beating ARGUE (54.9 and 66.3), ROUGE-L, and BERTScore. It is the only metric with positive correlation to grounding judgments on video.

• Human-computed MiRAGE validates the framework design. Human-annotated intrinsic claim judgments (which mirror InfoF1) averaged τ = 65.7 agreement with extrinsic quality judgments, showing that with human-level verification the metric captures quality preferences. Grounding judgments, by contrast, showed near-zero agreement (τ = 11.9) with holistic quality—meaning overall quality scores alone do not capture source groundedness.

• Subclaim-level beats sentence-level. Across all datasets and modalities, evaluating at the sentence level (as ALCE and ARGUE do) reduced agreement with human judgments. Atomic subclaims are the appropriate unit of evaluation.

• Surface metrics are brittle. ROUGE-L and BERTScore performed reasonably when high-quality human references existed (WikiVideo) but had near-zero or negative agreement on other datasets and failed entirely to capture grounding, with near-zero or negative correlation to grounding judgments.

• Automatic multimodal verification is the bottleneck. On the claim grounding benchmark, modality-specific models performed well only in their own modality, and the best omnimodal model (CLUE, 3B) reached 70.1% accuracy on audiovisual verification. Qwen3.5-27B hit 79.6% on video but the paper notes even this is well below what the human-computed results imply would be needed.

• The gap between human and automatic MiRAGE is attributable to verification quality, not framework design—reinforcing that future work should target multimodal claim verification.

Methodology in Plain English

The framework works like this. A generated text is broken into sentences, and each sentence is further broken into atomic, independently checkable statements called subclaims (e.g., "The fire started on Tuesday" and "The fire affected three buildings" from one sentence). Each subclaim is then tested against a knowledge source using an entailment-style scoring function—either an NLI model or an LLM judge—which returns a support score.

Two dimensions are scored. InfoF1 combines precision (what fraction of the system's subclaims are actually supported by the evidence) with recall (what fraction of the reference subclaims the system managed to cover). This can be done in a "reference" setting, where documents are first mapped to a text proxy like a human-written article, or a "collection" setting, where subclaims are verified directly against the raw documents, including videos. CiteF1 does the same for citations: precision asks whether each subclaim is supported by the specific source it cites, and recall asks whether reference subclaims that appear in the output are attributed to a source that genuinely supports them.

To validate the framework, the authors gathered human judgments on model outputs: holistic quality scores, claim-level factuality judgments, and grounding judgments against the videos. They then measured how well each automatic metric's ranking of systems agreed with the human ranking, using Kendall's τ. They also ran a separate claim grounding experiment, testing several audio, vision, and omnimodal models on their ability to verify a claim against text, audio, video, or combined evidence.

Why This Matters

Impact on research. The paper establishes that sentence-level and surface-level RAG metrics—the standard tools in the field—are the wrong unit for evaluating long-form generation from multimodal sources. It provides open-source implementations and a validated claim-centric alternative, giving the community a common measurement basis for comparing multimodal RAG systems.

Real-world applications:

• Newsroom and fact-checking tools that need to verify AI-generated summaries against source video footage.
• Accessibility and media monitoring systems that generate text descriptions from audio and video archives and must cite where each fact came from.
• Disaster response and crisis reporting, where firsthand video is often the primary source and accuracy of attribution matters operationally.
• Enterprise search over video repositories (meetings, lectures, surveillance), where generated answers must point to the correct clip.

Industry relevance. Any organization deploying RAG over video or audio—media platforms, educational technology, compliance and legal discovery—needs a way to measure whether outputs are grounded and correctly cited. This paper shows existing text-only pipelines do not transfer and offers a workable replacement, while also quantifying how much verification accuracy is currently lost.

Future Directions

• Improve multimodal claim verification. The best automatic verifiers achieve only modest accuracy on audiovisual grounding, and closing that gap is the primary open problem the paper identifies.
• Handle multi-video inference. Current models generally cannot fit multiple long videos in context, which limits both verification and the assessment of cross-source reasoning. Training and optimization work is needed here.
• Evaluate information refusal. MiRAGE does not capture cases where a system correctly declines to answer because information is absent from a large multimodal collection—a hard problem requiring either strong prior knowledge of the collection or near-perfect retrieval.
• Extend reference-free and weighted variants. The appendix outlines additional variants (single-source inference, importance weighting of subclaims and sources) that remain to be fully explored.

Target Audience

NLP and information retrieval researchers working on RAG evaluation; practitioners building or auditing multimodal generation systems over video and audio; benchmark and shared-task organizers who need metrics that transfer across modalities; and machine learning engineers who need to select evaluation metrics for production RAG pipelines.

Authors’ abstract

We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information online, RAG systems must integrate such media into generation. Yet, existing evaluation methods for RAG are largely text-centric and do not readily transfer to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE and compare it to multimodal variants of three prominent text-centric RAG metrics---ALCE, ARGUE, and RAGAS---finding that MiRAGE outperforms all three on text while being the only one to generalize to multimodal sources. We release open-source implementations and outline evaluation methods for multimodal RAG.