Hybrid-DMKG: A Hybrid Reasoning Framework over Dynamic Multimodal Knowledge Graphs for Multimodal Multihop QA with Knowledge Editing

AI summary

Overview

Research area: Multimodal knowledge editing (MKE), multimodal multihop question answering, dynamic knowledge graphs, and retrieval-augmented generation with large vision-language models (LVLMs). Submitted to arXiv (arXiv:2512.00881v1 [cs.AI], 30 Nov 2025) by authors from South China University of Technology, Guangdong University of Finance & Economics, Guangxi University, The Hong Kong Polytechnic University, and King's College London.

Technical level: Advanced. The paper combines knowledge-graph construction, cross-modal retrieval, relation-linking over triples, RAG prompting, and an LVLM-based arbitration module; familiarity with knowledge editing and LVLM pipelines helps.

Scope: The paper introduces MMQAKE, a benchmark for multimodal multihop QA under knowledge editing, and Hybrid-DMKG, a parameter-preserving framework that answers such questions with a dynamic multimodal knowledge graph plus parallel symbolic and RAG reasoning paths.

What This Paper Is About

Existing multimodal knowledge editing benchmarks judge whether a model produces the right final answer, but ignore whether the intermediate reasoning steps were correct and whether the model still answers correctly when the input image is swapped for a different picture of the same entity. The authors build a benchmark, MMQAKE, that scores every hop of a 2-to-5-hop text-and-image reasoning chain, adds visually rephrased images, and accepts valid answer aliases; they then build Hybrid-DMKG, a framework that stores edited multimodal facts in a dynamic knowledge graph and combines two different reasoning paths to answer the questions.

Key Contributions

1. MMQAKE benchmark. The paper presents MMQAKE as the first benchmark for multimodal multihop question answering with knowledge editing, extending the VLKEB benchmark. It requires reasoning over 2-to-5-hop factual chains that span text and images, scores each intermediate step, tests robustness with visually rephrased images of the same entity, and accepts Wikidata-based answer aliases.

2. Hybrid-DMKG framework. A parameter-preserving (no model-weight updates) framework built on a dynamic multimodal knowledge graph (DMKG) whose triples are (head, relation, tail) with images linked to entities. The graph supports dynamic insertion of edit quadruples (x, v, o, õ) and keeps both original and edited facts so the model can reason over either.

3. Hybrid reasoning with a reflective decision module. Answers are produced along two parallel paths — DMKG relation-linking prediction and RAG-enhanced reasoning inside an LVLM — and a background-reflective decision module then compares candidate answers using retrieved background context from the DMKG to choose the most credible one.

4. Systematic evaluation of MKE methods. The authors evaluate fine-tuning (Q-Former only and all parameters), MEND, SERAC, and IKE across three LVLM backbones, showing most struggle with multihop and cross-modal reasoning while Hybrid-DMKG performs best.

Main Findings

• Most existing MKE methods collapse on multihop multimodal questions. On MMQAKE, MEND performs worst, "failing to complete any multihop reasoning task," with M-Acc of 0.04% and H-Acc of 0.00% on BLIP-2 with original images. Larger models did not help: MiniGPT-4 (7.8B) frequently underperformed the smaller BLIP-2 (3.8B).

• IKE is the strongest baseline but loses accuracy with reasoning depth. IKE reached M-Acc / H-Acc of 16.64% / 6.16% on BLIP-2 with original images and 38.93% / 16.38% on LLaVA, but its H-Acc "declines significantly as the number of editing rounds increases."

• Hybrid-DMKG leads on both metrics and all three backbones. With original images: BLIP-2 47.55% M-Acc / 28.88% H-Acc; LLaVA 53.75% M-Acc / 29.90% H-Acc; MiniGPT-4 35.86% M-Acc / 24.73% H-Acc. With BLIP-2, its H-Acc surpasses IKE by 22.72% on original images.

• Visually rephrased images degrade everything, including the proposed method. Hybrid-DMKG scores 45.27% / 26.08% (BLIP-2), 51.27% / 26.16% (LLaVA), and 33.41% / 22.23% (MiniGPT-4) under rephrased images, but still outperforms all baselines; IKE with MiniGPT-4 showed "particularly pronounced declines" (9.86% M-Acc / 5.76% H-Acc).

• M-Acc is flat across hop counts; H-Acc is not. As shown in Figure 3, all models stay relatively stable under M-Acc regardless of hop count, which the authors say may reflect the metric only checking the final answer. Under H-Acc, performance declines as hops increase; on 4-hop and 5-hop questions Hybrid-DMKG achieves nearly double the baselines' accuracy, exceeding 5% on the hardest 5-hop setting while other methods typically remain below 2%.

• Both parallel reasoning paths and the decision module matter. In ablations, removing the Linking module hurt MiniGPT-4 more than removing RAG, with H-Acc dropping 13.93% under the rephrased-image setting (from 22.23% to 8.30%). Removing RAG leaves only link prediction, giving identical results for every backbone (28.13% / 21.50% on original images; 26.13% / 19.41% on rephrased images). Removing the Decision module also caused significant degradation, especially on rephrased images.

• H-Acc is the primary metric. Because a model can reach a correct final answer through an incorrect path, H-Acc requires every intermediate step to be correct and an instance is marked incorrect if any single step fails.

Methodology in Plain English

Task setup. Knowledge editing is written as a quadruple (x, v, o, õ): a text input x and image v are used to change a fact from o to õ. A multihop question then requires following a chain of facts where the object of one hop becomes the subject of the next.

Building and updating the graph. The framework maintains a multimodal knowledge graph where statements are triples and some entities carry an associated image. New edits are inserted into this graph, producing a dynamic graph that retains both old and new facts.

Breaking the question into steps. An LLM that is not fine-tuned applies a decomposition template to split a multihop question into a sequence of sub-questions. Sub-questions that need visual grounding use an [IMAGE] placeholder, and entities reused across sub-questions are replaced with an [ENT] token to keep them consistent.

Retrieving entities across modalities. For visual sub-questions, a cross-modal retrieval model encodes the sub-question together with the rephrased image, and separately encodes every candidate entity together with its image from the graph. The entity with the highest cosine similarity to the query vector is returned as the answer for that hop.

Answering reasoning sub-questions along two paths. The retrieved entity name is first normalized through a Wiki Linker module to its canonical form, then all triples attached to that entity are pulled from the graph.

• Relation-linking prediction: a fine-tuned relation extractor pulls the implicit relational keyword out of the query (for example, "country of birth"), the keyword is embedded with Sense2Vec, and the candidate relation with the highest cosine similarity is selected. If that similarity meets a threshold α, the corresponding object is the answer; otherwise the path returns nothing. (The numeric value of α is not reported.)
• RAG-enhanced reasoning: the top-K triples most relevant to the query are retrieved from the entity's knowledge set and placed into an answer prompt alongside the question and image, and the LVLM generates a candidate answer. (The value of K is not reported.)

Resolving disagreements. When the two paths disagree, the background-reflective decision module gathers background context for each candidate by again linking each candidate entity to the graph and expanding its neighborhood. The LVLM then sees the question, the image, both candidates with their background context, and a choice prompt, and produces the final answer. The same retrieve-then-reason procedure is repeated at each hop.

Training and evaluation details. The relation extractor was trained for 10 epochs with learning rate 2e-5, batch size 128, and the AdamW optimizer, on a relational-keyword dataset of 10,216 items (average question length 8.80, average labels 1.43), split 6:3:1 into training, development, and test sets. Backbones were BLIP-2 (3.8B), LLaVA-1.5 (7B), and MiniGPT-4 (7.8B). Baselines were fine-tuning (Q-Former only or all parameters), MEND, SERAC, and IKE. Each multihop question in MMQAKE was augmented with three paraphrased questions generated via the ChatGPT API, and answer alias sets were built from Wikidata references. Code is released at https://github.com/YuanLi95/Hybrid-DMKG.

Dataset statistics as reported (Table 2): per-hop edit counts of 1,278 (2-hop), 1,278 (3-hop), 1,238 (4-hop), 1,193 (5-hop), and 1,110; 11,773 total sub-questions; and an average of 9.49 answer aliases.

Why This Matters

Impact on research. The paper argues that scoring only final answers can hide reasoning failures — a model can land on the right answer via stale or wrong intermediate facts — and that ignoring visual rephrasing and answer aliases produces unfair or misleading evaluations. MMQAKE adds step-level scoring, visually rephrased inputs, and alias-aware matching, giving knowledge-editing researchers a stricter target; Hybrid-DMKG further shows that structured external graphs plus RAG can outperform weight-editing methods without touching model parameters.

Real-world applications:

• Updating assistant or search systems when facts about people, places, or organizations change, without retraining the underlying model.
• Multimodal question answering in which users supply a photo and ask a multi-step question that depends on recently corrected knowledge.
• Robust image-based retrieval and identification when the same entity appears in different photos (lighting, angle, framing).
• Knowledge-base maintenance pipelines where edits must propagate consistently through chained facts rather than being applied in isolation.

Industry relevance. Because Hybrid-DMKG is parameter-preserving, it points to a deployment pattern where knowledge is corrected in an external editable store instead of through repeated fine-tuning — attractive where models are large, edits are frequent, or forgetting previously learned behavior is costly. The benchmark also gives teams a way to measure whether an edit actually propagates through multi-step, cross-modal queries rather than just being echoed back on a single-hop probe.

Future Directions

• Temporal and event-based knowledge. The authors state they plan to extend MMQAKE to support dynamic knowledge updates that incorporate temporal and event-based information, addressing facts that change over time.
• Beyond factoid QA. They aim to address open-ended questions rather than only factoid questions with short, alias-matchable answers.
• End-to-end reasoning without predefined sub-questions. They plan to explore multihop reasoning that does not rely on a predefined decomposition into sub-questions, removing the dependency on the decomposition template.
• Costs and thresholds remain unquantified. The paper reports accuracies but not the specific values of the similarity threshold α, the retrieval budget K, the number of graph entities, or the latency and storage cost of maintaining the dynamic graph and running two reasoning paths plus a decision call — leaving open how the framework scales and what it costs at runtime.

Target Audience

Researchers and practitioners in knowledge editing, multimodal question answering, and retrieval-augmented generation; engineers building systems that must correct model knowledge without retraining; and benchmark designers interested in step-level and robustness-aware evaluation. Readers should be comfortable with knowledge-graph triples, cross-modal retrieval, and vision-language model prompting; the paper's benchmark tables and ablations are readable for those focused on evaluation methodology rather than graph internals

Authors’ abstract

Multimodal Knowledge Editing (MKE) extends traditional knowledge editing to settings involving both textual and visual modalities. However, existing MKE benchmarks primarily assess final answer correctness while neglecting the quality of intermediate reasoning and robustness to visually rephrased inputs. To address this limitation, we introduce MMQAKE, the first benchmark for multimodal multihop question answering with knowledge editing. MMQAKE evaluates (1) a model's ability to reason over 2-5-hop factual chains that span both text and images, including performance at each intermediate step, and (2) robustness to visually rephrased inputs in multihop questions. Our evaluation shows that current MKE methods often struggle to consistently update and reason over multimodal reasoning chains after knowledge edits. To overcome these challenges, we propose Hybrid-DMKG, a hybrid reasoning framework built on a dynamic multimodal knowledge graph (DMKG) to enable accurate multihop reasoning over updated multimodal knowledge. Hybrid-DMKG first uses a large language model to decompose multimodal multihop questions into sequential sub-questions, then applies a multimodal retrieval model to locate updated facts by jointly encoding each sub-question with candidate entities and their associated images. For answer inference, a hybrid reasoning module operates over the DMKG via two parallel paths: (1) relation linking prediction, and (2) RAG reasoning with large vision-language models. A decision module aggregates evidence from both paths to select the most credible answer. Experimental results on MMQAKE show that Hybrid-DMKG significantly outperforms existing MKE approaches, achieving higher accuracy and improved robustness to knowledge updates.