VideoLucy: Deep Memory Backtracking for Long Video Understanding

AI summary

Overview

Research area: Computer vision and multimodal large language models, specifically agent-based systems for long-form video question answering.

Technical level: Advanced. The paper assumes familiarity with vision-language models, LLM agent pipelines, retrieval, and video benchmarks, though the core idea is explained through an intuitive analogy to human memory.

Scope: The paper proposes VideoLucy, an agent framework that answers questions about very long videos by progressively "backtracking" through a hierarchy of memories from coarse overviews to frame-level detail, and releases a new benchmark (EgoMem) to test that capability.

What This Paper Is About

Existing agent-based video systems describe videos by captioning sparsely sampled, isolated frames, which loses the temporal context of consecutive frames and discards details when sampling is sparse. VideoLucy instead builds a layered memory of the whole video—like a person recalling a past event from a vague impression down to a specific detail—and iteratively digs deeper only where a question requires it. The paper also introduces EgoMem, a benchmark of 42 videos averaging 6.33 hours and 504 question-answer pairs, designed to test cross-temporal event understanding and perception of fleeting visual details in extremely long videos.

Key Contributions

1. A hierarchical memory structure with progressive granularity. The video is divided into clips, and memory detail is controlled by choosing how many clips K the video is split into (at K=1 the memory is a single overview of the whole video; at K=N it is a per-frame description). Three explicit levels are defined: long-range coarse memory, short-range fine memory, and frame-level ultra-fine memory. As memory deepens, the temporal scope shrinks while the number of descriptions (and detail) grows.

2. An agent-based iterative backtracking mechanism. Four prompt-engineered agents—Captioning (converts clips to text, the system's "eye"), Localization (picks the single most question-relevant time period), Instruction (identifies missing question-relevant information and writes a guiding caption prompt), and Answering (decides whether the current memory permits a confident answer)—drive a loop that expands memory in both depth and breadth until a confident answer can be given. A sparse coarse-memory initialization filters the initial memory list to question-relevant periods to limit cost.

3. The EgoMem benchmark. Built on EgoLife video resources, EgoMem contains 42 videos averaging 6.33 hours in length and 504 manually annotated questions, organized as six event-understanding question types plus a detail-perception task, with evidence descriptions for the annotations.

4. Strong empirical results with open-source models only. VideoLucy uses Qwen-2.5-VL-7B for captioning and DeepSeek-R1 for text reasoning, avoiding proprietary APIs, and reports state-of-the-art results on Video-MME, LVBench, MLVU, and EgoMem.

Main Findings

• Video-MME (without subtitles): VideoLucy scores 78.6 (short), 72.1 (medium), 66.8 (long), 72.5 average. The paper reports this as 8.5% higher on average than the previous best agent system, MemVid (64.0). On the long split it reports the best result among open-source MLLMs, ahead of AdaReTaKe-72B (65.0) by 1.8%, and on par with Gemini 1.5 pro. For reference, the table lists GPT-4o-20240513 at 71.9 average and Gemini 1.5 pro at 75.0 average.

• LVBench (ultra-long video): VideoLucy reaches 58.8% overall, with 54.3 (ER), 59.8 (EU), 75.6 (KIR), 51.7 (TG), 55.9 (Rea), and 49.1 (Sum). The paper states this is 5.5% above the previous leaderboard best (AdaReTaKe-72B at 53.3) and a 9.9% improvement over GPT-4o-20241120 (48.9). Key information retrieval (75.6%) is highlighted as far above all prior models.

• MLVU: VideoLucy achieves 76.1 M-Avg, compared with 74.7 for VideoChat-Flash-7B, 72.3 for Oryx-1.5-32B, 68.9 for LinVT-7B, 66.4 for LLaVA-OV-72B, 64.6 for GPT-4o, and 49.2 for GPT-4V.

• EgoMem: VideoLucy scores 58.7 (Event), 54.8 (Detail), and 56.7 average. The paper reports this as 10.3% above VideoChat-Flash-7B (46.4 avg), the best existing model on the benchmark. Most tested open-source MLLMs cluster between roughly 31 and 36 average—for example InternVL2.5-78B at 35.5 and Qwen2.5-VL-72B at 33.9—which the authors characterize as only slightly better than random guessing.

• Needle-in-a-Video-Haystack: With 10 long videos (400s to 4000s), five 10-second inserted clips each, and 4 questions per clip (20 questions per video), VideoLucy's performance is reported as significantly better than leading models and nearly unaffected by video length.

• Information richness and relevance increase during backtracking: Measured by average Shannon entropy of memory text descriptions and by an LLM acting as a relevance evaluator, both quantities rise steadily as the system backtracks.

• Depth and iteration ablations: Comparing answers from a rough video summary only, coarse memory, fine-grained memory, and ultra-fine memory access shows that going deeper improves performance, with the frame-level ultra-fine memory best. Setting the maximum iteration count to 5 gives the best performance and is used as the default.

• Qualitative comparison: Using a long video of the movie Lucy, VideoLucy answered cross-temporal event questions and fleeting-detail questions that the compared models (Qwen2.5-VL, InternVL3, VideoChat-Flash) handled inaccurately or incompletely, and it exposes the step-by-step reasoning behind each answer.

• Not reported: The paper does not give the numeric values of the temporal scopes T_c, T_f, and T_uf in the main text (it states they differ per benchmark and are given in the appendix), and no runtime, token cost, or monetary cost figures are reported in the provided content.

Methodology in Plain English

The system first watches the whole video in large chunks and writes a short text description of each chunk. This is the "coarse memory"—cheap to produce because few chunks are needed. When a question arrives, a localization agent points to the chunk most likely to contain the answer, and an instruction agent reads the current memory and works out what information is still missing. A captioning agent then re-describes that chunk with the missing information in mind, and also splits the chunk into smaller pieces and describes each one, producing a deeper, more detailed memory layer.

An answering agent then looks at the updated memory and asks: can I answer confidently? If yes, it answers. If not, the loop repeats, picking another relevant chunk and drilling deeper. The loop stops when the answer is confident or a maximum number of iterations (set to 5) is reached. The result is a memory that starts as a fuzzy impression of the entire video and sharpens only where the question demands it, avoiding the cost of captioning every frame.

Why This Matters

Impact on research. The paper argues that the two dominant weaknesses of agent-based video systems—frame-level reasoning that ignores temporal context, and sparse sampling that loses information—can be addressed by structuring memory hierarchically instead of sampling more densely. Treating memory design, rather than model scale, as the lever for long-video performance is the paper's distinctive position, and EgoMem adds a benchmark explicitly targeting hours-long video where current MLLMs score only slightly above random guessing.

Real-world applications.

• Reviewing hours of first-person lifelogging or wearable-camera footage for specific moments and events.
• Searching surveillance or security recordings for a described event or object appearing briefly.
• Navigating long recorded lectures, meetings, or industrial inspection footage to answer specific questions.
• Querying long-form media archives (film, sports, broadcast) for cross-temporal narrative or detail questions.

Industry relevance. VideoLucy is built entirely on open-source models (Qwen-2.5-VL-7B and DeepSeek-R1), so the paper positions the approach as reproducible and low-cost rather than dependent on proprietary APIs—relevant for teams that cannot send video to closed services. The paper also states that code and dataset will be made publicly available at videolucy.github.io.

Future Directions

• Tuning and generalizing the temporal scopes: T_c, T_f, and T_uf are set per benchmark in the appendix; how to set or adapt them automatically for unseen video lengths is left open.
• Cost and latency characterization: The iterative loop calls multiple agents repeatedly; the paper reports accuracy but not compute, token, or wall-clock costs, which would matter for deployment.
• Scaling the memory structure: Whether more than three hierarchy levels, or learned (rather than fixed) drill-down decisions, yield further gains is not explored.
• Broadening the benchmark: EgoMem is built on EgoLife first-person daily recordings; extending evaluation to other domains such as surveillance, film, or instructional video is a natural next step.
• Failure analysis: The paper reports aggregate accuracy but does not characterize which question types remain hardest or where backtracking terminates incorrectly.

Target Audience

Researchers and engineers working on video-language models, LLM agent systems, and long-context multimodal reasoning; practitioners building video search, lifelogging, or surveillance analytics who need to handle hour-scale video on a limited budget; and benchmark designers interested in evaluation protocols for cross-temporal and fine-grained video understanding.

Authors’ abstract

Recent studies have shown that agent-based systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as a promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the cost of dense frame-level captioning, they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limitations, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, VideoLucy employs a hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. Through an agent-based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information is gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding. EgoMem is designed to comprehensively evaluate a model's ability to understand complex events that unfold over time and capture fine-grained details in extremely long videos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, VideoLucy significantly outperforms state-of-the-art methods on multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly at https://videolucy.github.io