MoDora: Tree-Based Semi-Structured Document Analysis System

AI summary

Overview

• Research area: cs.IR — document understanding and retrieval-augmented question answering over semi-structured documents (mixed text, tables, charts, images, and supplements such as headers, footers, sidebars, and page numbers).
• Technical level: Advanced. The paper combines OCR pipelines, MLLM prompting, hierarchical tree construction, summarization, and multi-stage retrieval.
• Scope (one sentence): MoDora is an LLM-powered system that aggregates OCR-parsed document elements into layout-aware components, organizes them into a Component-Correlation Tree, and retrieves evidence with a question-type-aware strategy to answer natural-language questions over semi-structured documents.

What This Paper Is About

Semi-structured documents interleave tables, charts, hierarchical paragraphs, and other elements in irregular layouts, and answering natural-language questions over them requires linking evidence across scattered regions and pages. Existing methods fall short because OCR output is fragmented and stripped of context, existing representations do not capture hierarchical or layout-specific structure, and multi-region evidence is hard to retrieve and align. MoDora's goal is to build representations and retrieval mechanisms that let models jointly reason over the correct paragraphs, tables, and charts needed for an answer.

Key Contributions

1. MoDora system: a semi-structured document analysis system that supports diverse document layouts and handles both semantic and location-based questions. The code is released at https://github.com/weAIDB/MoDora.
2. Local-alignment aggregation: a strategy that converts OCR-parsed elements into self-contained components, combined with MLLM-based hierarchy detection for titles and template-based structured semantic extraction for non-text elements (tables, charts, images).
3. Component-Correlation Tree (CCTree): a hierarchical representation that captures inter-component relationships and layout-aware distinctions (separating main content from supplements such as sidebars and footnotes), built with a bottom-up cascade summarization process that propagates concise summaries upward.
4. Tree-based retrieval strategy: a question-type-aware retrieval approach combining LLM-guided node selection, embedding-based fallback for missed evidence, and MLLM-based cross-modal verification to validate selected nodes.

Main Findings

• Reported accuracy gain: the abstract and contribution list state that MoDora outperforms baselines by 5.97%–61.07% in accuracy. The specific benchmark names, dataset sizes, and per-benchmark numbers are not reported in the provided excerpt of the paper.
• Prevalence of structured elements: from an analysis of over one million real-world documents (collected from sources such as Scihub and PDF drive), over 77% contain at least one table, chart, or paragraph title; 61% include at least one table and 40% contain at least one chart.
• Existing methods fail for different reasons: content extraction methods (e.g., QUEST, EVAPORATE) discard document structure; structure extraction methods (e.g., ZenDB, DocAgent) lack fine-grained region modeling or robust structure inference; programmatic methods (e.g., Palimpzest) need manually written domain-specific programs; end-to-end models (e.g., LayoutLMv3, DocFormer, UDOP, DocOwl2, GPT-5) miss fine-grained textual details and cross-element relations; and RAG methods (e.g., TextRAG, SV-RAG, M3DocRAG) lack structural alignment.
• Failure of ZenDB on a worked example: ZenDB flattens Table 1 into plain text under Chapter 3, misidentifies the document hierarchy (nesting Chapter 3 under Chapter 2), and retrieves overly redundant content, so it cannot answer the example question.
• Failure of GPT-5 as a vision-language model: it hallucinates the answer 3.8 by analyzing only the table image and overlooking the surrounding paragraph indicating Experiment 3 was conducted in winter.
• Failure of GPT-5 as a RAG pipeline: it retrieves a sentence about how the average feather score was computed but fails to align the retrieved Table 1 in HTML-rendered form, producing the incorrect value 3.26 instead of the correct 3.20.
• ZenDB hierarchy error rate: the pilot study reports that ZenDB's error rate for hierarchy analysis approaches 50%, attributed to unreliable visual feature clustering and weak generalization to non-digital documents.
• Pre-trained model limits: models like LayoutLMv3 are constrained by model scale and maximum sequence length (e.g., 512 for text tokenization in LayoutLMv3) and by insufficient high-level document structure modeling.

Methodology in Plain English

MoDora assumes OCR output (for example from PaddleOCR) as input, so it works uniformly on digital-native and rasterized documents and retains typographic cues such as font size and indentation. The pipeline has three modules.

1. Document preprocessing. Fragmented OCR elements are merged into components using heuristic rules: a title plus the paragraphs up to the next title form one component; tables, charts, and images are paired with a preceding or following title (or given a default title) into a component; and each type of supplement on each page (headers, footers, sidebars, page numbers) becomes its own component. Locations are recorded as page indices and bounding box coordinates. Title hierarchy is detected by cropping the document regions of all titles, concatenating them vertically, and prompting an MLLM (e.g., GPT-5) with both text and image to assign each title a level. Non-text components are enriched by prompting an MLLM (e.g., Qwen2.5-VL) to produce a (title, metadata, data) triplet, using different few-shot templates for different element types.

2. Tree construction. Components become nodes of the CCTree, extended with a unique node index and a children list. Three edge types are defined: text-to-text (parent from the higher title level), text-to-other (a text component is parent of adjacent tables/charts/images), and independent supplementary edges (supplements attach to a virtual root rather than the main content). Construction happens in two stages: main body components are linked by reading order and title level using a stack-based algorithm, and supplementary components are attached under separate header, footer, sidebar, and number branches. A bottom-up summarization step propagates subtree summaries upward, with summary length constrained by an empirical formula to balance informativeness and redundancy.

3. Question answering over the tree. The system parses the question for locational and semantic cues. Location-based questions (for example, "top of page 1") are answered by layout-based grid partitioning over spatial metadata. Semantic questions use LLM-guided pruning: a forward search over the CCTree where the LLM selects nodes based on node index and metadata, followed by embedding search as a fallback to recover evidence in unselected subtrees, and an MLLM backward verification step to remove false positives. The final evidence set plus the tree structure is passed to an MLLM to generate the answer.

Why This Matters

• Research impact: the paper reframes semi-structured document QA as a structural modeling problem rather than a chunking or image-reading problem, and it introduces a reusable tree representation (CCTree) plus a retrieval protocol that others can build on. The pilot study quantifies where existing method families break down across four question types: text-only, hybrid data, structural hierarchy, and location-based.
• Real-world applications:
  • Scientific report analysis, where conclusions depend on linking prose about experimental setup to tables and charts on other pages.
  • Financial statements, where figures in tables must be interpreted alongside narrative disclosure.
  • Technical manuals, where procedures, diagrams, and specification tables are interleaved.
  • Any long document collection (the paper cites scientific reports, financial statements, and technical manuals, and its corpus draws from Scihub and PDF drive) that must be queried in natural language rather than through manually written programs.
• Industry relevance: the system targets practical document QA deployments where documents are rasterized or lack clean native PDF structure, and it explicitly handles the case where no domain-specific program is available, unlike Palimpzest-style approaches.

Future Directions

• Benchmark transparency: the excerpt does not report the benchmark names, dataset sizes, or per-benchmark results behind the 5.97%–61.07% accuracy improvement; reproducing and auditing those numbers requires the full experiments section.
• Complex multi-hop questions: the paper states that MoDora handles four representative question types and that more complex multi-hop questions (involving value aggregations and filtering) can be decomposed into combinations of those types using existing query decomposition techniques — whether that composition works in practice is left open.
• Robustness of the hierarchy step: since ZenDB's hierarchy error rate approaches 50%, a natural question is how sensitive MoDora's own MLLM-based title-level detection is to unusual layouts, and how errors propagate up the CCTree.
• Dependence on external models: preprocessing relies on OCR (e.g., PaddleOCR) and MLLM calls (e.g., GPT-5, Qwen2.5-VL) for hierarchy detection, information enrichment, node selection, and verification; the cost, latency, and behavior of the system under weaker or cheaper models are not addressed in the available content.

Target Audience

Researchers and practitioners in document AI, information retrieval, and retrieval-augmented generation who work with mixed-content documents. It is most valuable to readers already comfortable with OCR pipelines, LLM/MLLM prompting, and tree-based retrieval, and least suited to beginners seeking an introduction to document parsing, since the paper assumes existing OCR and MLLM infrastructure and does not focus on parsing itself.

Authors’ abstract

Semi-structured documents integrate diverse interleaved data elements (e.g., tables, charts, hierarchical paragraphs) arranged in various and often irregular layouts. These documents are widely observed across domains and account for a large portion of real-world data. However, existing methods struggle to support natural language question answering over these documents due to three main technical challenges: (1) The elements extracted by techniques like OCR are often fragmented and stripped of their original semantic context, making them inadequate for analysis. (2) Existing approaches lack effective representations to capture hierarchical structures within documents (e.g., associating tables with nested chapter titles) and to preserve layout-specific distinctions (e.g., differentiating sidebars from main content). (3) Answering questions often requires retrieving and aligning relevant information scattered across multiple regions or pages, such as linking a descriptive paragraph to table cells located elsewhere in the document. To address these issues, we propose MoDora, an LLM-powered system for semi-structured document analysis. First, we adopt a local-alignment aggregation strategy to convert OCR-parsed elements into layout-aware components, and conduct type-specific information extraction for components with hierarchical titles or non-text elements. Second, we design the Component-Correlation Tree (CCTree) to hierarchically organize components, explicitly modeling inter-component relations and layout distinctions through a bottom-up cascade summarization process. Finally, we propose a question-type-aware retrieval strategy that supports (1) layout-based grid partitioning for location-based retrieval and (2) LLM-guided pruning for semantic-based retrieval. Experiments show MoDora outperforms baselines by 5.97%-61.07% in accuracy. The code is at https://github.com/weAIDB/MoDora.