M$^3$KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation

AI summary

Overview

Research area: Natural Language Processing, specifically Retrieval-Augmented Generation (RAG) for multimodal large language models (MLLMs), combined with multimodal knowledge graphs covering audio, video, and text.

Technical level: Advanced. The paper assumes familiarity with RAG pipelines, knowledge graph construction, multimodal embedding spaces, and grounding models.

Scope: The paper proposes M³KG-RAG, an end-to-end framework that builds a multi-hop multimodal knowledge graph from raw video/audio/text corpora and retrieves only query-aligned, answer-supporting evidence for audio-visual question answering.

What This Paper Is About

Multimodal RAG systems that connect MLLMs to external knowledge still struggle in the audio-visual domain because existing multimodal knowledge graphs (MMKGs) mostly cover image–text pairs and are largely single-hop, so they miss temporal and causal relations across modalities. In addition, most retrieval relies purely on similarity search in a shared embedding space, which pulls in off-topic or redundant facts that do not actually help answer the question. The goal of this work is to build a multi-hop, modality-aware knowledge graph and a retrieval method that filters knowledge down to only what is both grounded in the query and useful for generating the answer.

Key Contributions

1. M³KG-RAG framework: An end-to-end graph-augmented multimodal RAG framework that integrates a multi-hop MMKG with retrieval to enhance audio-visual reasoning in MLLMs.

2. Three-step multi-agent MMKG construction pipeline: A lightweight collaborative multi-agent stack (rewriter, extractor, normalizer, searcher, selector, refiner, and inspector) that transforms raw multimodal corpora into a multi-hop MMKG (M³KG) through (i) Context-Enriched Triplet Extraction, (ii) Knowledge Grounding, and (iii) Context-Aware Description Refinement, plus a Self-Reflection Loop for quality control.

3. GRASP (Grounded Retrieval And Selective Pruning): A retrieval component that uses off-the-shelf multimodal grounding models to discard graph elements absent from the query and a lightweight LLM to prune triplets that do not contribute to answering, retaining only query-relevant, answer-useful subgraphs.

4. Broad empirical validation: Experiments across Audio QA, Video QA, and Audio-Visual QA benchmarks showing consistent gains over strong RAG baselines, including on a commercial MLLM.

Main Findings

• Large gains on Audio QA: On AudioCaps-QA, M³KG-RAG scored 53.23 Model-as-Judge (M.J.) with VideoLLaMA2 versus 44.60 for the strongest baseline (VAT-KG), and 60.77 with Qwen2.5-Omni versus 51.30 for VAT-KG and 49.00 for the no-retrieval setting.

• Gains on Video QA: On the VideoChatGPT (VCGPT) benchmark, M³KG-RAG reached 39.92 with VideoLLaMA2 (versus 39.42 for VAT-KG and 39.09 without retrieval) and 44.35 with Qwen2.5-Omni (versus 43.50 for VAT-KG and 42.21 without retrieval). These deltas are smaller than on the other benchmarks.

• Largest improvement on Audio-Visual QA: On VALOR, M³KG-RAG scored 44.67 with Qwen2.5-Omni versus 35.44 for VAT-KG and 32.42 with no retrieval, and 29.25 with VideoLLaMA2 versus 28.30 for VAT-KG.

• Baselines are inconsistent: The paper reports that text-KG with naïve RAG (Wikidata) yields weak or even negative deltas, and image–text KGs (VTKG, M²ConceptBase) only partially help and occasionally degrade results.

• Win-rate preference favors M³KG-RAG across all benchmarks and criteria: On AudioCaps-QA overall, M³KG-RAG won 84.8% against the no-retrieval baseline (15.2%), 86.3% against Wikidata, 78.8% against VTKG, 79.0% against M²ConceptBase, and 74.4% against VAT-KG. On VALOR, overall win rates ranged from 58.2% (vs. VAT-KG) to 61.7% (vs. M²ConceptBase).

• Both components are complementary and necessary: Ablation on VALOR with Qwen2.5-Omni showed 36.62 with neither component, 40.91 with Modality-Wise Retrieval only, 36.96 with GRASP only, and 44.67 with both.

• Works on a commercial model too: With GPT-4o, M³KG-RAG scored 59.17 (AudioCaps-QA), 53.05 (VCGPT), and 56.53 (VALOR), versus 57.70, 51.49, and 55.86 for VAT-KG and 56.74, 49.68, and 46.02 for no retrieval.

• Qualitative improvements in answer specificity: Examples include correcting "restaurant" to "a gathering or a party at someone's house," identifying a dodgeball scene, correcting a hallucinated keyboard player to two bagpipes and two drums, and linking buzzing audio to a running hair dryer instead of a speaker.

Methodology in Plain English

The authors start with a raw corpus of aligned text, audio, and visual samples. A lightweight multi-agent pipeline built on a single backbone LLM (Qwen3-8B) turns this into a knowledge graph in three stages. First, a rewriter agent enriches generic captions using crawled titles and descriptions, and an extractor agent pulls out subject–relation–object triplets. Second, a normalizer strips modifiers to get canonical entity names, and a searcher looks them up in open knowledge bases such as Wikipedia and Wiktionary, with an LLM fallback when nothing is found. Third, a selector picks the description that best fits the caption's context, and a refiner rewrites it to match the original entity phrasing. An inspector agent runs a self-reflection loop, assigning a plausibility score and sending problematic outputs back for re-generation. Every triplet is linked to at least one audio or visual item, guaranteeing full graph coverage.

At query time, the system avoids the "modality gap" problem by retrieving within the query's own modality first: video queries are embedded with InternVL2 and audio queries with CLAP, then searched against a FAISS index using L2 distance. The top k=5 nearest items are kept, filtered by a distance threshold tau, and then lifted to their connected triplets to form an initial subgraph. GRASP then applies two filters: GroundingDINO checks entity presence across four uniformly sampled video frames for visual grounding, and a Text-to-Audio Grounding model scores each triplet (converted into a natural sentence) against the query audio. Triplets below the presence thresholds are discarded, and a lightweight LLM applies a conservative keep-or-drop mask to remove knowledge that does not support the answer. The surviving subgraph is concatenated with the query, including entity–description pairs, and passed to the MLLM. Thresholds were set per benchmark: tau=3.0 and eta_a=0.5 for AudioCaps-QA, tau=0.15 and eta_v=1.5 for VCGPT, and tau=4.5 and eta_av=1.2 for VALOR. All experiments used a single NVIDIA H100 GPU.

Why This Matters

Impact on research: The work argues that multi-hop structure and fine-grained grounding-based pruning are jointly necessary for multimodal RAG, and that similarity search alone is insufficient once retrieval leaves the text modality. It positions itself against VAT-KG, whose graph is described as largely single-hop and whose RAG relies mainly on similarity-based search.

Real-world applications:

• Audio-visual assistants that must answer questions about video content with synchronized sound, such as identifying instruments, animals, or events.
• Accessibility tools that generate faithful descriptions of audio-visual media for users who cannot see or hear it.
• Media archiving and content tagging, where query-relevant entity and relation extraction from mixed media is needed at scale.
• Surveillance or industrial monitoring scenarios involving joint audio and visual streams, given the authors' affiliations with Hanwha Systems and funding from the Korea Research Institute for Defense Technology Planning and Advancement.

Industry relevance: The framework is built on off-the-shelf components (Qwen3-8B, InternVL2, CLAP, GroundingDINO, FAISS) and a single H100 GPU, indicating a practical, reproducible stack. The paper also reports results with GPT-4o, showing the method remains effective even for a high-capacity commercial model, which is relevant for deployment decisions. The project website is listed as https://kuai-lab.github.io/cvpr2026m3kgrag/.

Future Directions

• Closing the Video QA gap: Gains on the VCGPT benchmark were much smaller than on Audio QA and Audio-Visual QA, suggesting video-only retrieval and pruning may need further refinement.

• Extending beyond the truncated limitation discussion: The supplementary material states it discusses limitations and directions for improving robustness, but those details are not included in the provided content.

• Hyperparameter sensitivity and computational cost: The supplementary material promises hyperparameter sensitivity analysis and ablations over key components with their computational cost, but the reported values are not available in the provided text.

• Scaling coverage of the knowledge graph: The pipeline relies on Wikipedia, Wiktionary, an LLM callback, and crawled YouTube titles/descriptions; how well it generalizes to corpora without such metadata is an open question the provided content does not answer.

Target Audience

Researchers and engineers working on retrieval-augmented generation, multimodal large language models, and knowledge graph construction. The paper is most useful for those already familiar with RAG pipelines and multimodal grounding models who want to understand how multi-hop graph structure and grounding-based pruning can improve audio-visual reasoning. Practitioners building audio-visual QA or media understanding systems will find the implementation details (backbone models, thresholds, k value, hardware) directly actionable.

Authors’ abstract

Retrieval-Augmented Generation (RAG) has recently been extended to multimodal settings, connecting multimodal large language models (MLLMs) with vast corpora of external knowledge such as multimodal knowledge graphs (MMKGs). Despite their recent success, multimodal RAG in the audio-visual domain remains challenging due to 1) limited modality coverage and multi-hop connectivity of existing MMKGs, and 2) retrieval based solely on similarity in a shared multimodal embedding space, which fails to filter out off-topic or redundant knowledge. To address these limitations, we propose M$^3$KG-RAG, a Multi-hop Multimodal Knowledge Graph-enhanced RAG that retrieves query-aligned audio-visual knowledge from MMKGs, improving reasoning depth and answer faithfulness in MLLMs. Specifically, we devise a lightweight multi-agent pipeline to construct multi-hop MMKG (M$^3$KG), which contains context-enriched triplets of multimodal entities, enabling modality-wise retrieval based on input queries. Furthermore, we introduce GRASP (Grounded Retrieval And Selective Pruning), which ensures precise entity grounding to the query, evaluates answer-supporting relevance, and prunes redundant context to retain only knowledge essential for response generation. Extensive experiments across diverse multimodal benchmarks demonstrate that M$^3$KG-RAG significantly enhances MLLMs' multimodal reasoning and grounding over existing approaches. Project website: https://kuai-lab.github.io/cvpr2026m3kgrag/