SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding

AI summary

Overview

• Research area: Natural Language Processing, specifically multimodal document understanding and agentic multimodal large language model (MLLM) systems.
• Technical level: Intermediate. The paper assumes familiarity with MLLMs, retrieval-augmented generation (RAG), and multi-agent pipelines, but the three-level architecture is explained explicitly enough for readers with a general machine learning background.
• Scope in one sentence: The paper introduces SlideAgent, an MLLM-based agentic framework that builds hierarchical global-, page-, and element-level knowledge over multi-page visual documents such as slide decks, then retrieves and reasons over that knowledge to answer user queries.

What This Paper Is About

Multi-page visual documents (manuals, brochures, presentations, posters) convey information through layout, colors, icons, and cross-slide references, not just text. Current MLLMs tend to process each page holistically, struggle with domain-specific visual semantics and spatial reasoning, and often depend on clean metadata (figure locations, hierarchy tags, embedded text layers) that is missing or corrupted in real-world PDFs, scanned copies, or screenshots. The goal of SlideAgent is to answer fine-grained questions over these documents without relying on such metadata, by decomposing reasoning into three specialized levels.

Key Contributions

1. A three-level hierarchical agentic framework. SlideAgent uses specialized agents at the global (document-wide themes), page (page-specific features and cross-page relations), and element (fine-grained components such as charts, figures, and text blocks) levels, mirroring a human information processing model.
2. A two-stage design separating query-agnostic knowledge from query-specific retrieval. During knowledge construction the framework parses layout and builds a structured knowledge base that does not depend on the query; during inference it retrieves only the query-relevant knowledge and activates only the agents needed.
3. Metadata-free fine-grained grounding. The element agent integrates text detection, layout detection, and element classification with graph-based post-processing to merge fragmented text spans, capturing bounding boxes and element types directly from page images.
4. Query classification and subquery generation for agent orchestration. An orchestrator routes each query into one of four predefined categories (global understanding, fact-based direct queries, multi-hop reasoning, layout and visual relationships), defaulting to an "unknown" category that activates all agents, and expands short queries into subqueries targeting key entities.

Main Findings

• Improvements over base MLLMs: SlideAgent improves accuracy by +7.9% for proprietary models and +9.8% for open-source models (from the abstract); in the tables this appears as +7.9 overall on SlideVQA for GPT-4o and +9.8 overall on SlideVQA for InternVL3-8B.
• Proprietary results (Table 2): With GPT-4o as base, SlideAgent reaches 84.9 overall / 80.4 numeric / 90.5 F1 on SlideVQA (GPT-4o: 77.0 / 72.1 / 84.0), 70.9 / 82.5 / 66.2 on TechSlides (GPT-4o: 63.4 / 78.3 / 53.9), and 85.5 / 85.9 / 79.6 on FinSlides (GPT-4o: 80.0 / 80.8 / 62.1). Reported improvements are +7.9, +8.3, +6.5, +7.5, +4.2, +12.3, +5.5, +5.0, and +17.5 across these metrics.
• Open-source results (Table 4): With InternVL3-8B as base, SlideAgent scores 72.7 / 68.2 / 79.4 on SlideVQA (base: 63.0 / 56.5 / 74.1), 63.1 / 78.0 / 61.7 on TechSlides (base: 55.4 / 57.5 / 54.4), and 63.3 / 62.8 / 68.3 on FinSlides (base: 49.8 / 49.5 / 64.3). Improvements are +9.8, +11.7, +5.4, +7.7, +20.5, +2.3, +13.5, +13.3, +4.0.
• Single-page results (Table 3): On InfoVQA, SlideAgent scores 79.6 overall / 69.9 numeric / 94.1 F1, versus GPT-4o at 69.0 / 59.3 / 90.5 and ViDoRAG at 71.2 / 60.5 / 90.7 (improvements of +10.6, +10.5, +3.6).
• Beating stronger raw models: Although GPT-4o (77.0% overall on SlideVQA) lags Gemini-2.5-Flash (83.8%), SlideAgent's structured pipeline reaches 84.9% overall, closing that gap. SlideAgent achieves the best or second-best performance among proprietary models; for open-source models it is better across all evaluated models except the Qwen2.5-VL family.
• Biggest gains by query type (Figure 4): Multi-hop reasoning improves by 9.8% (67.4% to 77.2%), visual/layout reasoning by 7.7% (66.7% to 74.4%), and fact-based direct queries by only 2.1%.
• Ground-truth page setting: When the correct pages are supplied directly, eliminating retrieval noise, SlideAgent still improves over the base model by +7.7% overall and +12.5% numeric on SlideVQA.
• Error analysis (Figure 5): Of 40 annotated SlideVQA failure cases, only OCR Error and Tiny Visuals are directly attributable to parsing failures, together 5/40 cases (12.5%). The dominant categories are Ambiguous Question (22.5%), Answer Mislocation (17.5%), and Valid Alternative Answer (15.0%).
• Knowledge improves retrieval (Figure 6): Subqueries and page knowledge from SlideAgent boost text-based retrievers the most: SFR gains +6.4 MRR and +8.5 nDCG@1, and BM25 gains +4.9 MRR. Multimodal retrievers gain less: COLPALI +0.8 MRR but +9.7 Recall@3, VisRAG +3.7% MRR, and SigLIP2 +1.3% MRR.
• Ablations (Figure 7): Removing the page agent causes the steepest drop (-6.3 overall / -9.5 numeric for GPT-4o; -8.8 overall for InternVL3-8B). Removing the element agent costs -4.6 overall for GPT-4o and -6.3 for InternVL3-8B. Removing the global agent causes the smallest drop (-2.8% GPT-4o; -3.7% InternVL3-8B). Removing subquery generation under retrieval costs -5.0 for GPT-4o and -11.3 for InternVL3-8B, versus -2.9 to -4.9 with ground-truth pages.
• Qualitative example (Figure 8): For a query without global or page cues, all agents activate. The global agent nominates pages, the page agent identifies Page 4 ("Wealth Management–The Cause"), and the element agent parses the flowchart on that page, following the directed edge to return the answer "Business under-performance" with explicit provenance.

Methodology in Plain English

SlideAgent treats a multi-page document as a set of pages and a user query as something to be answered by reasoning over relevant visual and textual elements. It works in two stages.

Knowledge construction (query-agnostic). Working top-down, a global agent samples the first three pages to produce initial document-level knowledge covering the overall summary, objectives, and narrative flow. A page agent then processes each page sequentially, conditioned on that page's visual content, the initial global knowledge, and the knowledge produced for the preceding page, which creates a bridge between adjacent slides. After all pages are processed, the global agent performs a single-pass fieldwise rewrite, regenerating each global field from the complete page-level evidence, using the initial global knowledge only as a weak prior, so later pages can overwrite earlier hypotheses and reduce bias toward the first pages. Finally, an element agent decomposes each page into elements via a layout parsing pipeline that records page index, verbatim text, bounding box coordinates, and element type, integrating text detection, layout detection, and element classification, plus post-processing with a graph-based merging algorithm to rejoin fragmented text blocks. Each element gets knowledge describing its semantic role, functional purpose, and relation to the slide.

Inference (query-specific). An agent orchestrator first classifies the query into a category that determines which agents run, avoiding unnecessary computation and noise. Global understanding queries activate only the global agent; fact-based direct queries trigger page and element agents; multi-hop reasoning and layout/visual relationship queries use different combinations; an unrecognized query defaults to an "unknown" category that activates all agents. The short original query is expanded into subqueries targeting key entities, and the query plus subqueries are concatenated to retrieve the top pages and top elements and their associated knowledge. Each agent produces a hypothesis with its reasoning (global, page, and element hypotheses). If all agents agree according to answer matching, or only one agent is activated, that answer is used directly; otherwise an answer synthesizer combines the agents' reasoning and the visuals from retrieved pages into the final answer.

Why This Matters

Impact on research. The paper argues that page-level retrieval alone is not enough and provides systematic evidence that element-level reasoning produces measurable gains. It reframes documents as retrieval-then-reasoning problems and demonstrates that structured knowledge generated by agents also improves the retrievers themselves, not just the final answer. The error analysis showing that most failures come from question ambiguity, retrieval drift, and annotation subjectivity, rather than parsing, is a useful calibration for where future effort should go.

Real-world applications.

• Financial analysis: analysts extracting key metrics from earnings decks, where the datasets include FinSlides from the REAL-MM-RAG suite and where color conventions encode categories (red for losses, green for gains).
• Technical documentation review: engineers inspecting multi-page technical specifications, evaluated with TechSlides, to avoid implementation errors.
• Education and study material generation: educators and students working with lecture slides and multi-page presentations, where inaccuracies can propagate at scale.
• Metadata-free document processing: handling screenshots, scans of physical documents, flattened PDF exports, and PDFs generated by software that strips or does not preserve document structure, which the framework supports because it parses only visual images.

Industry relevance. The work was done in part at J.P. Morgan AI Research, and the paper carries an explicit J.P. Morgan disclaimer and acknowledgements to J.P. Morgan AI Research staff, indicating a direct financial-industry motivation. The framework is model-agnostic, so organizations can apply it on top of both proprietary and open-source backbones, and the finding that cheaper text-based retrievers rival multimodal LLM-based methods has practical cost implications.

Future Directions

• Improve parsing robustness on low-contrast layouts, which the authors list as a future direction and which relates to the OCR Error and Tiny Visuals failure modes in the error analysis.
• Model inter-element relations via graphs, extending the current element-level knowledge, which captures each element's role and its relation to the slide, into richer relationships among elements.
• Replace the first-N-pages heuristic with content-aware page selection for the initial global knowledge, addressing the current reliance on sampling the first three pages.
• Extend SlideAgent to multi-turn scenarios, moving beyond single-query question answering.
• Broaden element-level evaluation. The limitations section notes that element-level grounding was assessed on a manually annotated subset of SlideVQA questions and that element boundaries may vary across OCR and layout-parsing tools, so broader benchmarks with element-level annotations are needed. The authors also note that they primarily adopt text-based retrieval for efficiency and that multimodal or domain-specific retrieval strategies remain open.

Target Audience

Researchers and practitioners working on multimodal document understanding, visual question answering over slide decks and long PDFs, retrieval-augmented generation, and multi-agent MLLM systems. It is particularly relevant to engineers and applied scientists in finance, technical documentation, and education who need to query real-world, metadata-free multi-page visual documents, and to readers interested in how agent decomposition at multiple granularities affects reasoning accuracy.

Authors’ abstract

Multi-page visual documents such as manuals, brochures, presentations, and posters convey key information through layout, colors, icons, and cross-slide references. While multimodal large language models (MLLMs) offer opportunities in document understanding, current systems struggle with complex, multi-page visual documents, particularly in fine-grained reasoning over elements and pages. We introduce SlideAgent, a versatile agentic framework for understanding multi-modal, multi-page, and multi-layout documents, especially slide decks. SlideAgent employs specialized agents and decomposes reasoning into three specialized levels--global, page, and element--to construct a structured, query-agnostic representation that captures both overarching themes and detailed visual or textual cues. During inference, SlideAgent selectively activates specialized agents for multi-level reasoning and integrates their outputs into coherent, context-aware answers. Extensive experiments show that SlideAgent significantly improves accuracy over both proprietary (+7.9%) and open-source models (+9.8%).