MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering

AI summary

Overview

Research area: Computer vision and natural language processing, specifically multimodal retrieval-augmented generation (RAG), long-form visual question answering, and source attribution (citation-grounded text generation).

Technical level: Advanced. The paper assumes familiarity with vision-language models, RAG pipelines, retrieval metrics, and entailment-based evaluation.

Scope: The paper introduces MAVIS, a benchmark of 157K visual question-answering instances (1K human-annotated for evaluation) that measures whether vision-language models can answer image-based questions with long-form responses backed by sentence-level citations to multimodal evidence.

What This Paper Is About

AI systems that generate long answers often make things up, and existing solutions for citing sources only work with plain text—they ignore images entirely. MAVIS fills this gap by building the first benchmark that tests whether a model can look at a user's question and image, search for both text and image documents, and write a detailed answer where every sentence is traced back to supporting evidence. The goal is to make AI answers verifiable in deliberately visual, real-world settings where an image carries information that words alone cannot.

Key Contributions

1. The MAVIS benchmark itself. A dataset of 157,586 visual QA instances (1,000 of them human-annotated for evaluation) where each answer is annotated with fact-level citations pointing to specific multimodal documents. The dataset contains 81,173 unique questions, 67,140 images, and averages 5.9 text documents and 3.1 image documents per question.

2. A three-dimensional automatic evaluation framework. Fine-grained metrics for informativeness (completeness + relevance), groundedness (recall + precision), and fluency (MAUVE), validated against human judgment with Pearson correlations between 0.733 and 0.903.

3. A novel "Knowledge Extraction" (KE) step. Before generating the final answer, the model independently extracts factual content from each retrieved document without seeing the user's question or image. This removes the contextual bias that causes models to hallucinate when interpreting images.

4. Empirical characterization of modality bias in LVLMs. Systematic experiments showing that vision-language models systematically over-rely on text documents and under-utilize image evidence, especially in mixed-modality retrieval settings.

Main Findings

• Multimodal RAG beats unimodal RAG on coverage and fluency, but not on grounding. Models retrieving from a combined text-image database produce more complete and more fluent answers than text-only or image-only retrieval. For example, GPT-4o with Multi-RAG scores 44.4 informativeness F1 versus 37.0 for Text-RAG and 40.2 for Image-RAG.

• Image documents are cited far less accurately than text documents. Groundedness F1 for image retrieval ranges from 17.0% to 53.1% across models, while text retrieval reaches 66.7% to 73.4%. Adding text documents to a multimodal pool makes image grounding even worse.

• A trade-off exists between informativeness and groundedness across prompting strategies. Chain-of-Thought prompting with Qwen2.5VL raises informativeness by 6.3% but drops groundedness by 3.8%; with GPT-4o, CoT raises groundedness by 7.2% but loses 11.7% informativeness. No prompting method improves both at once.

• Knowledge Extraction resolves part of that trade-off. Adding the KE step improves groundedness by up to 4.3% and informativeness by up to 4.2% simultaneously without sacrificing the other metric.

• KE disproportionately fixes image grounding. In multimodal retrieval, image groundedness jumps from 26.8% to 56.0% and image document utilization rises from 25.5% to 39.3% when KE is applied. Text-related metrics barely change.

• A single text document can collapse attention to images. Adding just one text document to five image documents drops image utilization from 58.2% to 26.25% in GPT-4o's output, revealing severe text dominance in multimodal reasoning.

• Retrieving more documents helps coverage but hurts naturalness. Informativeness rises with more retrieved documents (up to 4.2% improvement at 25 documents), groundedness peaks around 15 documents, and fluency steadily declines.

• Multi-query retrieval improves coverage at the cost of fluency stability. Generating several search queries per instance gives more diverse evidence and higher informativeness than a single query fetching the same total number of documents, but produces less consistent prose.

• Data construction filters work. 89% of MAVIS questions genuinely seek information (versus 49% of raw Reddit posts), and 89% depend on the attached image (versus 68% of raw posts).

Methodology in Plain English

The researchers built the dataset by mining 6 million Reddit posts that contained both images and long comment threads, then keeping only posts whose titles were genuine questions answerable with a substantial written response. To make sure the image actually mattered, they had four language models try to answer each question without seeing the image; if any model succeeded, the instance was discarded. This left 157,586 visual QA pairs.

For each answer, they used GPT-4.1 to split the response into atomic facts, then searched for supporting evidence using those facts as queries—pulling web pages and images from Google Programmable Search and a large web-crawled text corpus. A retrieval and filtering pipeline, using entailment models like Qwen3-8B for text and SkyworkVLReward-8B for images, removed irrelevant documents, leaving 930K text and 500K image documents.

They then sampled 1,000 instances across different topic domains and had humans verify that each fact was relevant to the question and genuinely supported by its documents. The resulting test set contains roughly 3K relevant atomic facts backed by about 5.1K documents.

For evaluation, they tasked models with three steps: generate search queries from the question and image, retrieve top documents using the MM-Embed multimodal retriever (chosen after benchmarking it against CLIP-DPR, UniVL-DR, MARVEL, and others), and produce a final long-form answer with bracketed citations. GPT-4.1 judged informativeness and groundedness, while MAUVE measured fluency. A separate human study of 600 model answers confirmed the GPT-4.1 scores matched human ratings closely.

Why This Matters

Impact on research: MAVIS extends citation-based attribution, previously a text-only problem, into multimodal territory. It provides the first standardized testbed for questions that require both visual understanding and verifiable evidence, and it surfaces a concrete, measurable failure mode—text dominance—that future multimodal RAG systems must address.

Real-world applications:

• Accessibility tools for blind and low-vision users, where a photo plus a spoken question must yield a trustworthy, sourced explanation of what is shown.
• Technical and repair assistance, such as identifying a mechanical component in a photo and citing diagrams or manuals that justify each step of the answer.
• Medical, legal, and educational assistants, where users need to check every claim against an original source before acting on it.
• E-commerce and customer support, where a customer uploads a picture of a defective product and expects a diagnosis backed by documentation rather than a confident guess.

Industry relevance: Any company deploying RAG-based assistants faces the same challenge MAVIS measures: producing answers that users can verify. The finding that models ignore image evidence when text is present has direct implications for product search, multimodal customer service, and any pipeline that mixes document types. The Knowledge Extraction technique is a lightweight, model-agnostic fix that can be added to existing systems without retraining, making it immediately practical for teams building citation-aware generation.

Future Directions

1. Weight-level optimization for attribution. The paper only evaluates prompting methods. Training or fine-tuning models specifically to cite multimodal evidence accurately remains unexplored and is the authors' explicit next step.

2. Scaling beyond text and images. Extending MAVIS to video, audio, and other modalities would let models ground answers in dynamic or temporal evidence, particularly useful for questions about motion, sequences, or events.

3. Mitigating text dominance more fundamentally. Knowledge Extraction helps, but the underlying tendency of LVLMs to favor text over images when both are available is not solved. Understanding and correcting this architectural or training-data bias is an open problem.

4. Breaking the informativeness–groundedness trade-off. No current method achieves both high coverage and high citation accuracy at once. Finding prompting or training strategies that avoid this zero-sum dynamic is a central challenge the benchmark now makes measurable.

Target Audience

This paper is most valuable to researchers working on retrieval-augmented generation, vision-language models, and attribution or factuality in generative AI. Practitioners building production RAG systems that mix text and image evidence will find the modality-bias findings and the Knowledge Extraction technique directly actionable. Benchmark designers and evaluation researchers will benefit from the metric validation methodology and the human-annotation protocol. Graduate students entering multimodal NLP or computer vision will find it a useful, well-structured entry point into the open problems of grounded long-form generation.

Authors’ abstract

Source attribution aims to enhance the reliability of AI-generated answers by including references for each statement, helping users validate the provided answers. However, existing work has primarily focused on text-only scenario and largely overlooked the role of multimodality. We introduce MAVIS, the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations. Our dataset comprises 157K visual QA instances, where each answer is annotated with fact-level citations referring to multimodal documents. We develop fine-grained automatic metrics along three dimensions of informativeness, groundedness, and fluency, and demonstrate their strong correlation with human judgments. Our key findings are threefold: (1) LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but they exhibit weaker groundedness for image documents than for text documents, a gap amplified in multimodal settings. (2) Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods. (3) Our proposed method highlights mitigating contextual bias in interpreting image documents as a crucial direction for future research.