MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models

AI summary

Overview

Research area: Multimodal large language models (LMMs), temporal reasoning benchmarks, and multimodal knowledge editing.

Technical level: Intermediate. Readers should be comfortable with benchmark design, multimodal model evaluation, exact-match scoring, and the general idea of knowledge editing, but the paper is written without heavy math.

Scope: The paper introduces MINED, a benchmark for probing how well large multimodal models handle time-sensitive factual knowledge, evaluates 15 models on it, and then tests whether knowledge editing can repair the gaps those models show.

What This Paper Is About

Large Multimodal Models store facts in their parameters, but those parameters are frozen at training time, so the models give outdated or hallucinated answers when asked about facts that change over time (for example, which club a footballer currently plays for). Existing temporal benchmarks are mostly text-only or statically designed, so they do not properly measure whether a multimodal model can detect temporal misalignment, refuse unanswerable dates, understand implicit time expressions, or reason over time. The paper builds MINED to fill that gap and then asks whether knowledge editing methods can update the missing time-sensitive knowledge in models that fail on the benchmark.

Key Contributions

1. The MINED benchmark. A benchmark of 2,104 time-sensitive knowledge samples spanning six fine-grained knowledge types, which evaluates temporal awareness along six dimensions (cognition, awareness, trustworthiness, understanding, reasoning, robustness) and 11 challenging tasks. Each knowledge instance is stored as a quadruple (subject S, hypernym H, property P, and a time-varying list of attribute values A).

2. A six-dimensional evaluation framework with prompt agreement. Four semantically equivalent prompts ("Question", "Generalization Question", "Image", "Generalization Image") are designed per knowledge instance, and the final score averages across these configurations to reduce sensitivity to prompt wording.

3. A 15-model evaluation. Widely used LMMs are evaluated on MINED: LLaVA-v1.5, Qwen-VL, mPLUG-Owl2, LLaVA-Next (Mistral-7B), LLaVA-OneVision, mPlug-Owl3, MiniCPM-V2.6, Qwen2-VL, InternVL2.5, Qwen2.5-VL, GPT-4.1, Kimi-Latest, Doubao-1.5-Vision-Pro, Gemini-2.5-Pro, and Seed-1.6-Vision.

4. An exploration of knowledge editing for time-sensitive knowledge. Parameter-modifying methods (FT-LLM, FT-VIS, MEND) and parameter-preserving methods (SERAC, IKE) are applied to LLaVA-v1.5 (7B) and Qwen-VL (7B) under single-editing and lifelong-editing settings.

Main Findings

• Highest average score. Gemini-2.5-Pro achieves the highest average CEM score of 63.07 across the 15 evaluated LMMs. Seed-1.6-Vision is next at 55.16, followed by GPT-4.1 at 51.82. At the bottom, LLaVA-v1.5 (7B) scores 15.85 and mPLUG-Owl2 (7B) scores 24.74.

• Timestamp-aware questions are easiest. Across the three cognition subtasks, LMMs perform best on Timestamp-Aware (T.S.A) queries relative to Time-Agnostic (T.A) and Temporal Interval-Aware (T.I.A). Even Gemini-2.5-Pro fails to recall approximately 15% of the knowledge.

• Misaligned context hurts, especially past misalignment. Compared with T.S.A results, performance drops when queries are accompanied by temporally misaligned context. Qwen2-VL I. (7B) shows declines of 43.84% on F.M.C and 56.43% on P.M.C. Closed-source models and larger open-source models are more robust; smaller open-source models degrade substantially.

• Future unanswerable dates are rejected more readily than past ones. Most LMMs reject questions with unanswerable past or future dates. Qwen2-VL I. (7B) reaches an average CEM of 99.64 and Qwen2.5-VL I. (7B) reaches 99.70 on refusal, which the authors attribute to instruction-tuning defenses. Note that the paper describes this as surprising because models still commit errors on the main factual subtasks.

• Implicit temporal concepts are a widespread failure. On the Implicit Temporal Concept (I.T.C) task, all models perform poorly, and the best model, Gemini-2.5-Pro, recalls 18.73% — under 20%.

• Ranking favors open-source, calculation favors closed-source. MiniCPM-V2.6 (8B) at 52.78 and InternVL2.5 (8B) at 61.73 lead the Ranking (R.K) task, while GPT-4.1 at 15.74 and Doubao-1.5-Vision-Pro at 18.52 fall below 20%. Within the Qwen2.5-VL I. series, ranking performance declines with size: 50.3 (3B) → 38.9 (7B) → 11.4 (72B), which the authors suggest may stem from overthinking. On Calculation (C.A), Gemini-2.5-Pro (76.54) and GPT-4.1 (59.57) lead.

• Adversarial robustness is weak. Qwen-VL (7B), LLaVA-Next M. (7B), and InternVL2.5 (8B) score 0.00 on the Adversarial Temporal Error (A.T.E) task, correcting no prior errors. Even Gemini-2.5-Pro corrects fewer than 40% of errors (39.58).

• Newer models tend to score higher. The average results show an approximate trend that more recently released models achieve better overall temporal awareness.

• Knowledge type matters. LMMs perform better on organization, company, and country leader queries, and worse on athletes and competition champions. Closed-source models outperform open-source variants on university president queries.

• Most model outputs are outdated or irrelevant. In the Time-Agnostic fine-grained analysis, LLaVA-v1.5 (7B) produces the latest value only 14.90% of the time, with 27.45% outdated and 57.65% irrelevant. Seed-1.6-Vision produces 21.57% latest and 64.31% outdated. Open-source models produce many irrelevant responses, whereas closed-source models reduce irrelevance but still produce many outdated answers.

• Single editing works well. Using knowledge editing to add time-sensitive knowledge that LLaVA-v1.5 (7B) and Qwen-VL (7B) originally missed, FT-LLM achieves an average of 97.2 on LLaVA-v1.5 (7B) and 91.3 on Qwen-VL (7B). SERAC is the weakest at 61.6 and 57.8 respectively. Except for SERAC, all methods perform excellently on the A.T.E task.

• Lifelong editing still degrades. For LLaVA-v1.5 (7B), FT-LLM falls to an average of 54.0 (a 43.2 drop), FT-VIS to 34.8 (a 51.8 drop), and SERAC to 51.2 (a 10.4 drop). SERAC's memory-based architecture mitigates catastrophic forgetting, and its A.T.E score improves by 12.6% under lifelong editing.

Methodology in Plain English

The researchers first built a source dataset. Two professional annotators collected entities from Wikipedia across six domains: country, sport, company, university, organization, and competition. GPT-4o was prompted to produce candidate entity lists, and annotators manually verified each candidate on Wikipedia, keeping only entities that are visual (representable by an image) and time-sensitive (their attributes change over time). Each kept entity is stored as a quadruple: the subject, its hypernym category, the property being asked about, and a list of attribute values with start and end dates.

To make sure a model is at least able to see the entity, the authors wrote five perception question templates and tested every data point on the 15 LMMs; if 10 of the models failed to identify the entity in the image, the sample was discarded. For each surviving entity they downloaded generalization images from Google using subject plus hypernym as the query, extracted CLIP features for the downloaded and original images, excluded pairs with similarity 1, and kept the top-1 result as the generalization image.

From these quadruples they generated task questions by writing templates per domain for each of the 11 tasks. Cognition has three subtasks (Time-Agnostic, Temporal Interval-Aware, Timestamp-Aware); Awareness has Future Misaligned Context and Past Misaligned Context, where GPT-4o generates a context describing the current or a past attribute to create a conflict with the query timestamp; Trustworthiness has Past Unanswerable Date and Future Unanswerable Date, built by moving outside the valid date range of the attribute list; Understanding has Implicit Temporal Concept, where explicit periods are replaced with references to historical events (for example, "when Jeff Bezos served as CEO of Amazon" for July 5, 1994, to July 5, 2021); Reasoning has Ranking and Calculation; and Robustness has Adversarial Temporal Error, applied only to samples that every model got wrong, using the prompt "Your answer to the original question is wrong."

Scoring uses Cover Exact Match (CEM), where a prediction counts as correct only if it is contained in the ground truth, and each dimension's score is the average CEM across its subtasks. Because prompt wording can change results, the authors average across four semantically equivalent prompt variants, a strategy they call Prompt Agreement.

Finally, for the editing experiments they selected LLaVA-v1.5 (7B) and Qwen-VL (7B) as the outdated models because of their poor benchmark performance. Editing samples were drawn from cases where CEM was not 100 across cognition, trustworthiness, understanding, reasoning, and robustness. They compared parameter-modifying methods (FT-LLM, FT-VIS, MEND) with parameter-preserving methods (SERAC, IKE) under single editing (weights restored after each edit) and lifelong editing (all edits applied before evaluation).

Why This Matters

Research impact. MINED shifts temporal evaluation from pure text into multimodal settings and adds dimensions that prior benchmarks such as TimeQA, MenatQA, TempReason, UnSeenTimeQA, EvolveBench, and LiveVQA do not jointly cover — notably trustworthiness (refusing unanswerable dates), implicit temporal concepts, and adversarial robustness. It also connects temporal evaluation to the knowledge-editing literature, showing that editing is a plausible repair path in the single-edit case but still lossy across many edits.

Real-world applications.

• Question answering assistants that answer factual questions about people, teams, and organizations whose status changes over time, where an outdated answer is a factual error.
• News and knowledge-base verification systems that must detect when retrieved context is temporally misaligned with the user's question.
• Hallucination and refusal systems, where a model must recognize that a date falls outside a valid range and decline rather than fabricate.
• Retrieval-augmented pipelines, since the paper's error analysis finds models often ignore a supplied context or draw an outdated value from memory even when the context is right.

Industry relevance. Any product built on a frozen multimodal model faces knowledge staleness. The finding that FT-LLM reaches 97.2 average CEM in single editing on LLaVA-v1.5 (7B) suggests targeted patching is viable, while the lifelong results (SERAC's 10.4 drop versus 43.2 to 51.8 drops for other methods) indicate that sustained updating remains an operational problem for deployed systems.

Future Directions

• Improve temporal consistency between internal knowledge and external context. The conclusion calls for methods that help models distinguish the temporal consistency of their own parameters from the context they are given, since the Awareness dimension is weak.

• Handle implicit temporal concepts. The Understanding dimension scores are low across all models, and even Gemini-2.5-Pro stays below 20% on I.T.C, so semantic comprehension and transformation of implicit time expressions is an open problem.

• Build stronger self-correction and adversarial robustness. The Robustness dimension shows several models scoring 0.00 on A.T.E and the best model below 40%, motivating new self-correction mechanisms.

• Extend beyond static images and beyond six domains. The stated limitations are that MINED covers only static visual data and not complex temporal dynamics such as video, and that highly specialized, time-critical fields such as law and medicine are left for future exploration.

• Close the lifelong editing gap. Single editing is effective, but performance drops substantially in lifelong editing, so making updates cumulative without forgetting is an unresolved technical hurdle.

Target Audience

Researchers and engineers working on multimodal large language models, temporal reasoning, factual knowledge grounding, hallucination and refusal behavior, and knowledge editing. The paper is also useful for benchmark designers who want a template for building multidimensional, domain-stratified evaluation sets, and for practitioners deciding whether knowledge editing is a practical way to keep a deployed multimodal model current.

Authors’ abstract

Large Multimodal Models (LMMs) encode rich factual knowledge via cross-modal pre-training, yet their static representations struggle to maintain an accurate understanding of time-sensitive factual knowledge. Existing benchmarks remain constrained by static designs, inadequately evaluating LMMs' ability to understand time-sensitive knowledge. To address this gap, we propose MINED, a comprehensive benchmark that evaluates temporal awareness along 6 key dimensions and 11 challenging tasks: cognition, awareness, trustworthiness, understanding, reasoning, and robustness. MINED is constructed from Wikipedia by two professional annotators, containing 2,104 time-sensitive knowledge samples spanning six knowledge types. Evaluating 15 widely used LMMs on MINED shows that Gemini-2.5-Pro achieves the highest average CEM score of 63.07, while most open-source LMMs still lack time understanding ability. Meanwhile, LMMs perform best on organization knowledge, whereas their performance is weakest on sport. To address these challenges, we investigate the feasibility of updating time-sensitive knowledge in LMMs through knowledge editing methods and observe that LMMs can effectively update knowledge via knowledge editing methods in single editing scenarios.