VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically long-video question answering and multi-turn conversational memory.

Technical level: Advanced. The paper assumes familiarity with video QA benchmarks, MLLM evaluation protocols, and LLM-based data construction pipelines, though the core idea is stated plainly enough for newcomers.

Scope: The paper introduces VideoMemBench, a benchmark of 326 single-video dialogues (2,476 turns) with explicitly typed cross-turn dependencies, to measure whether MLLMs can reuse information established earlier in a conversation about a video.

What This Paper Is About

MLLMs can now ingest hour-long videos and hold extended conversations about them, but existing video QA benchmarks ask questions that are independent of one another: nothing requires the model to carry information from an earlier turn into a later one. The authors define in-context video memory as the ability to retrieve and reuse both what the model has seen (earlier visual episodes) and what the model has said (its own earlier answers), and they build a benchmark that makes this ability mandatory by design. The goal is to measure how well current MLLMs actually do when the dialogue history is load-bearing rather than decorative.

Key Contributions

1. Defines in-context video memory as an explicit evaluation target, separating memory of what the model has seen (Episodic Dependency) from memory of what the model has said (Answer Dependency), and shows that existing benchmarks leave it unmeasured.
2. Builds VideoMemBench, where each sample pairs one video with a long multi-turn dialogue whose turns are linked by a typed dependency graph: 326 videos, 2,476 turns, of which 1,387 are dependent, with 1,794 typed dependencies (1,200 Episodic, 594 Answer). It supports multiple parents (348 multi-parent turns: 289 double-parent, 59 triple-parent) and long-range call-backs.
3. Enforces validity of every dependency through three requirements (necessity, non-leakage, visual grounding) and a construction pipeline with three LLM roles (planner, generator, judger) plus review of every retained dialogue by eight human experts.
4. Evaluates 25 open- and closed-source MLLMs, showing that the best model (Claude Opus 5) reaches only 45.7% overall against 86.1% for humans, and that accuracy falls as dependency distance, parent count, and dialogue position increase.

Main Findings

• Headline gap: Claude Opus 5 is the strongest of the 25 evaluated models with 45.7% overall (52.3% TurnAcc, 49.1% EpiDepAcc, 35.7% AnsDepAcc), while human annotators reach 86.1% overall (88.1% TurnAcc, 86.0% EpiDepAcc, 84.2% AnsDepAcc).
• Answer memory is harder than episodic memory: for closed-source models, AnsDepAcc is 16.6–22.9 percentage points below TurnAcc, compared with only 3.9 percentage points for humans. Part of this reflects chain crediting, where a turn is credited only if every turn along its Answer-dependency ancestry is also correct.
• Dependent turns trail independent turns: across the five models in Table 3, the mean of EpiDepAcc and AnsDepAcc is 13.1–17.3 percentage points below independent-turn accuracy.
• Closed-source ranking: after Claude Opus 5 (45.7%), Gemini 3.8 Flash scores 42.4%, GPT-5.6-sol 41.7%, Gemini 3.5 Flash 41.6%, GPT-5.5 40.5%, Claude Sonnet 5 40.2%, Gemini 3.1 Pro Preview 39.1%, and GPT-5.4 the lowest at 36.3%.
• One open-weight model is competitive: Kimi K3 scores 42.2% overall (50.5% TurnAcc, 49.0% EpiDepAcc, 27.1% AnsDepAcc), ranking third overall and second on TurnAcc and EpiDepAcc across all evaluated models.
• Most open-weight models fall below the weakest closed-source model: the runner-up open-weight model, Qwen3-VL-235B-A22B, scores 34.4% overall, 7.8 percentage points below Kimi K3; Qwen3.5-27B follows at 34.3%. The remaining open-weight models all score below GPT-5.4's 36.3%.
• Scale helps within families: among the evaluated dense Qwen3.5 models, overall score rises from 16.2% at 0.8B to 34.3% at 27B, and Qwen3-VL-32B outperforms Qwen2.5-VL-32B by 6.2 percentage points.
• Longer dependencies are harder: mean accuracy across all evaluated models drops from 30.2% at dependency distance 1 to 23.1% at distances of five or more turns. Gemini 3.8 Flash declines relatively steadily from 41.1% to 30.2%, while Qwen3.5-35B-A3B falls from 27.8% to 19.2% beyond distance 3.
• More parents are harder: accuracy declines monotonically with parent count. Kimi K3 falls from 40.8% with one parent to 29.8% with three; GPT-5.4 from 34.9% to 23.7%; Qwen3.5-27B from 30.1% to 17.3%.
• Later positions are harder, especially for dependent turns: Gemini 3.8 Flash's dependent-turn accuracy falls from 46.1% at turn 2 to 35.2% at turns 5 and beyond, while its independent-turn accuracy falls from 53.8% to 49.2%. Qwen3.5-27B shows the same pattern, 36.2% to 28.1% on dependent turns versus 46.1% to 40.3% on independent ones.
• Earlier errors compound: swapping in oracle histories (ground-truth prior answers) raises AnsDepAcc from 12.3% to 32.4% for Qwen3-VL-8B, 17.2% to 37.3% for Qwen3-VL-32B, and 29.7% to 47.2% for Gemini 3.5 Flash. Oracle accuracy still stays below independent-turn accuracy, so correct history alone is not sufficient.

Methodology in Plain English

The authors start from a pool of roughly 3.2k videos drawn from Video-MME, VSI-Bench, Video-MMMU, LongVideoBench, EgoLife, and YouTube (sports, talks and lectures, tutorials, film recaps, cooking), then screen them for recurring entities and events that can support cross-turn links. The final set spans 31 semantic domains across four categories: Lifestyle & Entertainment (130 videos), Knowledge & Education (80), Art & Literature (66), and Sports & Competition (50).

Before any question is written, a planner LLM watches sampled keyframes and produces a full dialogue specification: a video synopsis, dialogue guidelines (turns to build, capabilities to test), a directed acyclic dependency graph assigning each dependent turn one or more earlier parents typed as Episodic or Answer, and a per-turn intent with an evidence window. Deciding the structure first is what makes dependencies necessary rather than incidental.

A generator LLM then writes the dialogue turn by turn. For each turn it first identifies a verifiable visual fact in the evidence window to serve as the reference answer, then writes a question that asks for that fact. Dependent turns refer to earlier turns only through referring expressions (for example "the player identified in T1") without restating what those turns established.

A judger LLM applies three checks, each matching a stated requirement:

• Necessity (Req-1): remove the parent turns and confirm the question becomes unanswerable from the video and remaining dialogue.
• Non-leakage (Req-2): confirm the question does not partially restate a parent's scene description or disclosed answer.
• Visual grounding (Req-3): confirm frames from the annotated evidence window actually support the reference answer, and reject turns that semantically duplicate earlier ones.

Failed turns are regenerated with the judger's feedback; two consecutive failures trigger replanning of that portion of the dialogue. Completed dialogues are checked against the original plan, and eight human experts then review every surviving dialogue, revising or removing turns with incorrect answers, invalid dependencies, or leakage.

Evaluation protocol: all models receive the same fixed set of 128 frames sampled across the full video with timestamps, presented at the start of the conversation and kept accessible throughout. Questions arrive strictly in dialogue order with the model's own prior responses retained in the history, so the input at turn t is the frame sequence, then all prior (question, model answer) pairs, then the current question. GPT-5.4 mini grades open-ended responses and extracts predicted options for multiple-choice items. Frame-count ablations at 16, 32, and 64 frames are reported in Appendix E.1, and a separate human baseline evaluation is reported in Appendix D.

Why This Matters

Impact on research. The paper gives the video understanding community a way to measure something previously invisible: whether a model's dialogue history actually does any work. Because every dependency is verified for necessity and non-leakage, a low score cannot be explained away as "the question was too hard anyway" or "the answer was already in the prompt." The typed Episodic/Answer split also lets researchers attribute failures to memory of perception versus memory of self-generated text, and the chain-crediting metric formalizes error compounding across a conversation.

Real-world applications:

• Long-form video assistants that accompany a user through a film, match, or lecture where follow-up questions routinely refer back to earlier discussion.
• Sports and coaching review, where an analyst revisits a passage of play identified several exchanges earlier.
• Educational lecture and course Q&A, where a student's later question builds on a concept the model already explained.
• Egocentric and day-long recordings (the paper includes EgoLife footage), where recurring people and activities must be tracked across distant moments.
• Media archive and film-recap navigation, where a user drills into a scene referenced only by an earlier turn.

Industry relevance. Any product deploying a video-capable assistant in a conversational loop depends on this exact ability. The results are a direct warning: the models tested lose roughly 13 to 17 percentage points on dependent turns relative to independent ones, with answer-dependent accuracy dropping to 35.7% even for the best model. Vendors building on multi-turn video chat should expect degradation as conversations lengthen, and the finding that oracle histories substantially help but do not close the gap suggests that both better retrieval and better self-consistency are needed.

Future Directions

• Improve self-consistency, not just retrieval: oracle histories lift answer-dependent accuracy by 20.1 to 20.2 percentage points for the two Qwen3-VL models and 17.5 points for Gemini 3.5 Flash, yet remain below independent-turn accuracy. What mechanism beyond correct history is missing?
• Handle multi-parent and long-range dependencies explicitly: accuracy falls with parent count (down to 29.8% for Kimi K3 at three parents) and with distance (23.1% at five or more turns), and the paper reports a maximum chain depth of seven in the dataset. Methods that summarize, index, or re-retrieve dialogue state could target this.
• Reduce error compounding across Answer chains: the 16.6–22.9 percentage point gap between TurnAcc and AnsDepAcc for closed-source models points to upstream failures propagating downstream; training or prompting for stable commitment to earlier answers is an open problem.
• Extend the evaluation beyond the current frame budget: the main results use a fixed 128 frames, with 16/32/64-frame ablations in Appendix E.1; how memory behaviour changes with native long-context video input, and whether frame budgets interact with dependency distance, is not resolved here.

Target Audience

Researchers and engineers working on multimodal LLMs, long-video understanding, and conversational agents; benchmark designers who need a model for constructing verified multi-turn datasets; and product teams evaluating whether a video assistant can sustain realistic multi-turn dialogue. Readers new to the area will find the conceptual distinction between episodic and answer memory the most immediately useful takeaway; specialists will care most about the dependency-graph construction pipeline and the chain-crediting metric.

Authors’ abstract

Multimodal large language models (MLLMs) can now process hour-long videos, making extended multi-turn dialogue over a single video a natural mode of use. Sustaining such a dialogue requires more than answering each question in isolation: the model must retrieve and reuse information established earlier in the conversation, both what it has seen and what it has said, an ability we term in-context video memory. Existing video QA benchmarks leave this ability unmeasured: their questions are mutually independent, never reference one another, and each is answerable from the video alone, so nothing ever requires carrying information across turns. We introduce VideoMemBench, in which each sample pairs a single video with a long multi-turn dialogue, whose turns are connected by typed cross-turn dependencies: an Episodic Dependency refers back to a visual episode described in an earlier turn, probing memory of what the model has seen, while an Answer Dependency builds on the model's own earlier answer, probing memory of what the model has said. Every dependency is verified considering two properties: 1) necessity, meaning the question cannot be answered without its earlier turns, and 2) non-leakage, meaning the question does not restate the information from its earlier turns. To enforce these properties, we design a construction pipeline with three LLM roles: a planner, a generator, and a judger, with each dialogue reviewed by human experts. VideoMemBench comprises 326 videos with 2,476 turns, of which 1,387 depend on earlier turns, including multi-parent dependencies and long-range call-back. Among over 20 evaluated open- and closed-source MLLMs, Claude Opus 5 performs the best with an overall score of only 45.7%, highlighting the challenge of in-context video memory. Our dataset and evaluation are released.