VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understanding

AI summary

Overview

• Research area: Computer vision and multimodal large language models (MLLMs), specifically training-free agentic systems for long-form video question answering.
• Technical level: Advanced (assumes familiarity with MLLMs, ReAct-style agents, retrieval pipelines, and token-budget accounting).
• Scope: The paper proposes VideoARM, an agentic "observe–think–act–memorize" framework that reasons over a dynamically built three-tier memory instead of exhaustively preprocessing video, and evaluates it on Video-MME, LongVideoBench, EgoSchema, MLVU, and LVBench.

Paper details: by Yufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding, Zhenwei Shao, and Zhou Yu (corresponding author), School of Computer Science and Technology, Hangzhou Dianzi University, China, and Key Laboratory of Complex Systems Modeling and Simulation. arXiv:2512.12360v2 [cs.CV], dated 30 Mar 2026, licensed CC BY 4.0; code at https://milvlg.github.io/videoarm/.

What This Paper Is About

Long-form videos contain dense, evolving multimodal information that must be searched and reasoned over, but existing MLLM-based approaches either follow hand-crafted pipelines (LLoVi, VideoTree) or, like DVD, exhaustively caption every 10-second clip into a database before any question is asked. VideoARM's goal is to replace that static, token-heavy preprocessing with adaptive, query-guided reasoning, where an agent decides on the fly what to look at, records what it finds in a hierarchical memory, and spends only the tokens it needs.

Key Contributions

1. An agentic reasoning-over-hierarchical-memory paradigm (VideoARM) that disc

Authors’ abstract

Long-form video understanding remains challenging due to the extended temporal structure and dense multimodal cues. Despite recent progress, many existing approaches still rely on hand-crafted reasoning pipelines or employ token-consuming video preprocessing to guide MLLMs in autonomous reasoning. To overcome these limitations, we introduce VideoARM, an Agentic Reasoning-over-hierarchical-Memory paradigm for long-form video understanding. Instead of static, exhaustive preprocessing, VideoARM performs adaptive, on-the-fly agentic reasoning and memory construction. Specifically, VideoARM performs an adaptive and continuous loop of observing, thinking, acting, and memorizing, where a controller autonomously invokes tools to interpret the video in a coarse-to-fine manner, thereby substantially reducing token consumption. In parallel, a hierarchical multimodal memory continuously captures and updates multi-level clues throughout the operation of the agent, providing precise contextual information to support the controller in decision-making. Experiments on prevalent benchmarks demonstrate that VideoARM outperforms the state-of-the-art method, DVD, while significantly reducing token consumption for long-form videos.