Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding

AI summary

Overview

• Research area: Computer vision and multimodal large language models, specifically long-video understanding with retrieval-augmented generation (RAG).
• Technical level: Intermediate. The paper assumes familiarity with large video language models (LVLMs), retrieval-augmented generation, and graph-based knowledge representation, but its pipeline is described at a conceptual level.
• Scope: The paper proposes Vgent, a training-free graph-based retrieval-reasoning-augmented generation framework, and evaluates it on seven open-source LVLMs (2B to 7B) across three long-video benchmarks: MLVU, Video-MME, and LongVideoBench.

What This Paper Is About

Understanding hour-long videos is hard because the number of visual tokens grows very quickly: the paper notes that a 30-minute video can exceed 200K tokens, beyond most models' context limits. Existing solutions either sample frames sparsely (losing detail) or use retrieval-augmented generation (RAG), but video RAG tends to break temporal continuity by treating clips as independent documents and to pull in irrelevant "hard negative" clips that distract the model. Vgent's goal is to preserve relationships across video clips with a graph representation, then verify and aggregate the retrieved clips through an intermediate reasoning step before generating an answer.

Key Contributions

1. A graph-based RAG framework for long video. Video clips are represented as nodes in a graph and connected through shared entities extracted from the clips and their spoken content, which preserves semantic relationships and temporal dependencies across clips and enables more effective retrieval.
2. A structured reasoning stage to handle weak reasoning in LVLMs. After retrieval, the framework decomposes the question into subqueries whose answers are binary (yes/no) or numerical, uses them to verify and filter each retrieved clip, and then aggregates information across the surviving clips before generation.
3. Comprehensive evaluation across seven LVLMs and three benchmarks. Vgent is applied to InternVL2.5, Qwen2-VL, Qwen2.5-VL, LongVU, and LLaVA-Video at sizes from 2B to 7B, reporting 3.0%–5.4% improvements over base models on MLVU and outperforming state-of-the-art video RAG methods by 8.6%.
4. A query-independent, self-contained pipeline with efficiency gains. Graph construction is performed offline and does not depend on the question, so the same graph can be reused for multiple questions on the same video. The framework achieves a 1.73× speedup over Video-RAG on Video-MME and avoids reliance on proprietary LLMs for graph construction.

Main Findings

• Consistent gains on MLVU. In Table 1, Vgent improves LongVU (7B) by 5.4% (65.4 to 70.8), Qwen2.5-VL (7B) by 3.3% (68.8 to 72.1), Qwen2-VL (7B) by 4.6% (65.7 to 70.3), LLaVA-Video (7B) by 3.0% (69.5 to 72.5), Qwen2.5-VL (3B) by 4.2% (66.2 to 70.4), and InternVL2.5 (2B) by 4.4% (56.7 to 61.1).
• A 3B model surpasses its 7B counterpart. With Vgent, Qwen2.5-VL (3B) reaches 70.4% on MLVU, exceeding the 7B Qwen2.5-VL result of 68.8% and improving its own base model by 4.2%.
• Gains on Video-MME and LongVideoBench. On Video-MME, every base model improves: Qwen2.5-VL (7B) goes from 65.1 to 68.9 without subtitles (+3.8) and 71.1 to 74.3 with subtitles (+3.2); InternVL2.5 (2B) goes from 49.5 to 50.9 (+1.4) and 55.2 to 56.8 (+1.6); LongVU (7B) from 55.2 to 57.3 (+2.1) and 60.9 to 63.7 (+2.8). On LongVideoBench, Qwen2.5-VL (7B) improves from 56.0 to 59.7 (+3.7) and Qwen2.5-VL (3B) from 54.2 to 57.8 (+3.6).
• Reported Video-MME aggregate figures differ by section. The Table 1 caption states an overall improvement of 3.2% on Video-MME, while the main text states an average performance gain of 4.2% and a 5.4% gain in long-video scenarios.
• Vgent beats Video-RAG across three base models. In Table 2, Qwen2.5-VL (3B) with Video-RAG scores 62.2 / 60.3 / 65.1 (MLVU, Video-MME without and with subtitles) versus 70.4 / 63.0 / 69.6 with Vgent. LLaVA-Video (7B) with Video-RAG scores 71.3 / 64.8 / 70.0 versus 72.5 / 66.7 / 71.1 with Vgent. Qwen2.5-VL (7B) with Video-RAG scores 63.4 / 60.5 / 65.7 versus 72.1 / 68.9 / 74.3 with Vgent.
• The pipeline also surpasses proprietary RAG methods. The comparison lists VideoAgent at 44.4, LLoVi at 67.7, and DrVideo at 71.7, with Vgent's open-source configurations exceeding these.
• GraphRAG beats NaïveRAG. In the ablation on Qwen2.5-VL, NaïveRAG lowers MLVU performance from 68.8 to 65.4, while GraphRAG raises it to 69.5. The paper reports GraphRAG gives an average improvement of 2.9% over NaïveRAG, with a 4.1% gain on MLVU.
• Structured reasoning adds further gains. Adding structured reasoning to GraphRAG yields an additional 2.6% on MLVU (69.5 to 72.1) and 1.6% on Video-MME (72.7 to 74.3), an overall 3.4% average gain over the base model. Structured reasoning applied to NaïveRAG instead produces 68.6 on MLVU, below the 68.8 base, indicating the refinement depends on graph-based retrieval quality.
• Retrieval noise is a real bottleneck. The paper reports that in roughly 40% of failure cases the correct clip is retrieved but the model still answers incorrectly, and that in MLVU specifically, 44% of failures involve the correct clip being present in the retrieved set.
• More retrieved clips help up to a point. In the r-ablation on MLVU, overall accuracy rises from 63.2 at r=1, to 66.9 at r=2, 68.4 at r=3, 71.0 at r=4, peaks at 72.1 at r=5, and dips slightly to 71.9 at r=6. The Count subcategory rises most dramatically, from 25.7 at r=1 to 58.7 at r=4 and r=5.
• Efficiency. Per minute of video, Vgent requires 20.13 seconds offline for graph construction and 3.93 seconds online for retrieval, reasoning, and generation, compared with 20.81 seconds online for Video-RAG (no offline component) and 67.25 seconds for VideoAgent (no offline component). The graph is built once, giving a 1.73× speedup over Video-RAG on Video-MME's multi-question setting.
• Qualitative failure and fix. In the example shown in Figure 3, the model answers "No" to "Did I open the laptop?" because hard negatives—clips featuring an already-open laptop—cause hallucination. Structured subqueries such as "Is there a laptop open?" and "Is someone interacting with the laptop?" let the model verify the relevant clip and correctly infer that the laptop was opened.

Methodology in Plain English

Vgent is a four-stage, training-free pipeline:

1. Offline video graph construction. The video is sampled at 1.0 FPS and split into clips of K = 64 frames each. For every clip, the LVLM extracts key semantic entities (subjects, actions, scenes) from the clip together with its subtitles, producing entity-description pairs. A global set of unique entities is maintained: a newly extracted entity is merged into an existing one if their text-embedding similarity exceeds a threshold of τ = 0.7; otherwise it becomes a new entity. Whenever a clip's entity matches an existing entity, edges are added between that clip's node and all other nodes containing the same entity. This graph is query-independent and can be reused for multiple questions.
2. Graph-based retrieval. The LVLM extracts keywords from the user query. Each keyword is compared against entity descriptions, and nodes whose entity similarity exceeds θ = 0.5 are collected as candidates. Candidates are then re-ranked by similarity between query keywords and each node's entities, descriptions, and subtitles, and the top N = 20 clips are selected.
3. Structured reasoning. The LVLM generates structured subqueries from the question and keywords—questions designed to be answered with yes/no or a number. Each of the top-N clips is checked against these subqueries, and clips failing all subqueries are discarded. At most r = 5 clips survive.
4. Multimodal augmented generation. LVLM summaries of the subquery results, together with the refined clips, are fed as multimodal context to produce the final answer.

Embeddings for similarity computation come from BAAI/bge-large-en-v1.5. Spoken content is extracted with openai/whisper-large for MLVU, while Video-MME and LongVideoBench come with provided subtitles. All experiments run on A100 80G GPUs. Thresholds are kept the same across all three benchmarks.

Why This Matters

The work targets a practical bottleneck: as video grows longer, token-based and sparse-sampling approaches either exceed context windows or lose the temporal detail needed for multi-event reasoning. Vgent shows that structuring video content as an entity-linked graph, and then verifying retrievals before generation, materially improves open-source LVLMs without any training or proprietary API calls, and that a 3B model enhanced this way can outperform a 7B baseline.

Real-world applications:

• Web content and streaming media, where narratives and evolving context span long durations—explicitly named as a target domain in the paper.
• Life-logging from wearable cameras, where a user asks about events spread across a recorded day.
• Video archive and media-asset search, where a single query must be answered from footage spanning hours.
• Assistive and accessibility tools that need to answer natural-language questions about long recordings.

Industry relevance: Because graph construction is offline, query-independent, and reusable across multiple questions, the approach maps onto production settings where many queries are asked about the same long video (the paper notes Video-MME has three questions per video). Avoiding proprietary LLM APIs, and achieving 20.13 seconds of offline preprocessing plus 3.93 seconds per minute of video online, makes the pipeline cheaper and more deployable than agent-based alternatives such as VideoAgent at 67.25 seconds per minute of video.

Future Directions

• Addressing the source of remaining failures. The paper reports that about 40% of failures occur even when the correct clip is retrieved, and 44% on MLVU; how much of that residual error structured reasoning can remove is left open.
• Extending the graph approach beyond open-source LVLMs and the evaluated sizes. The evaluation covers 2B to 7B models; behavior with larger or different model families is not reported in the provided content.
• Reducing dependence on subtitles and audio. MLVU required Whisper transcription, while Video-MME and LongVideoBench supplied subtitles; performance in a purely visual, subtitle-free setting is not reported in the provided content.
• Details deferred to the appendix. Limitations are discussed in Appendix D, and category-level MLVU results, confidence-based refinement, ablations on the number of retrievals N, and the retrieval threshold τ are placed in Appendices A.1, A.2, A.5, and A.6 — none of these appendix results are included in the provided content, so their conclusions cannot be summarized here.

Target Audience

Researchers and practitioners working on long-video understanding, multimodal retrieval-augmented generation, and video-language models. It is most useful for readers who want to improve an existing open-source LVLM's long-video question answering without fine-tuning, and for engineers building retrieval pipelines over long video archives where cost, latency, and reuse across multiple queries matter. Readers new to RAG or graph-based retrieval will need background reading, while specialists will find the graph-construction and structured-verification design most relevant.

Authors’ abstract

Understanding and reasoning over long videos pose significant challenges for large video language models (LVLMs) due to the difficulty in processing intensive video tokens beyond context window and retaining long-term sequential information. Retrieval-Augmented Generation (RAG) has demonstrated effectiveness in processing long context for Large Language Models (LLMs); however, applying RAG to long video faces challenges such as disrupted temporal dependencies and inclusion of irrelevant information that can hinder accurate reasoning. To address these limitations, we propose Vgent, a novel graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding. Our approach introduces two key innovations: (i) It represents videos by structured graphs with semantic relationships across video clips preserved to improve retrieval effectiveness. (ii) It introduces an intermediate reasoning step to mitigate the reasoning limitation of LVLMs, which leverages structured verification to reduce retrieval noise and facilitate the explicit aggregation of relevant information across clips, resulting in more accurate and context-aware responses. We comprehensively evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks. Our approach yielded an overall performance improvement of $3.0\%\sim 5.4\%$ over base models on MLVU, and outperformed state-of-the-art video RAG methods by $8.6\%$. Our code is publicly available at https://xiaoqian-shen.github.io/Vgent.