Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation

AI summary

Overview

• Research area: Computer vision and document AI, especially scientific-document parsing and visual accessibility.
• Technical level: Intermediate.
• Scope: The paper introduces a benchmark for producing a structured, ordered parse of a scientific-document page—including its text, layout, captions, and descriptions of visual elements—and a model adapter intended to improve spatial accuracy.

The paper reports a dataset of 2,402 documents, 125,218 layout elements, and 11,338 visual instances. Its page count is inconsistent: the abstract reports 18,163 continuous pages, while the main text and appendix report 18,164.

What This Paper Is About

Document systems often handle layout, text recognition, reading order, and image description as separate tasks. That can leave them unable to produce one coherent account of a page—for example, correctly connecting a figure to its caption and describing the right figure. The authors create Read-Parse-Describe to evaluate this combined task and propose SARA, an adapter that gives a vision-language model more spatial guidance.

Key Contributions

1. Read-Parse-Describe benchmark: Defines a shared output format in which each page element has a location, category, reading-order position, relevant text or table structure, and, when applicable, a caption link and alternative-text description.
2. Scientific-document dataset: Collects open-access ACM papers, retaining the first eight pages of each paper or the full paper when it is shorter. The dataset covers varied page elements and visual types, including tables, diagrams, screenshots, photographs, and charts.
3. SARA adapter: Introduces a plug-in method that reuses intermediate visual features and aligns them with the model’s image crops before adding them to the visual tokens used for generation.
4. Joint evaluation and experiments: Evaluates end-to-end models and pipeline parsers on the same schema, reporting layout, reading-order, OCR, and visual-description measures.

Main Findings

• The task exposes gaps hidden by separate evaluations. A system may perform well on OCR or localization yet fail to recover reading order, connect captions to the correct objects, or provide alternative text. Pipeline parsers, for example, can do well on low-level fields but may not produce alternative text under the evaluated schema.
• SARA improved several structural measures for Qwen3.5-2B. Compared with the paper’s full fine-tuning baseline, Qwen3.5-2B + SARA achieved F1 of 0.9189 versus 0.7336, mIoU of 0.8919 versus 0.8350, OCR accuracy of 0.9636 versus 0.9408, and reading-order score of 0.9774 versus 0.9628. The SARA run trained 83.99M parameters while keeping the base model frozen.
• SARA’s effects depend on the model and metric. For Gemma4-E4B, SARA improved F1 and mIoU relative to full fine-tuning, but its reading-order and description scores were slightly lower. The paper therefore does not present SARA as uniformly best on every measure.
• Different systems led on different results. Fine-tuned Dolphin-v2 had the highest reported localization F1, 0.9826, and reading-order score, 0.9857. Fine-tuned DeepSeek-OCR had the highest METEOR, 0.3998, while ChatGPT-5.6 had the highest BERTScore, 0.9004.
• The adapter’s components worked best together. In the Qwen3.5-2B ablation, grid alignment plus the projector achieved F1 of 0.9189, compared with 0.7541 for grid alignment alone and 0.6830 for the projector alone.
• Alternative-text labels received a review process. Gemini 3 Pro generated initial descriptions, and Gemini 3 Flash scored them. Of the 1,408 descriptions scoring below 4, the paper reports that human refinement raised the score-5 rate from 66.2% to 98.1%. A manual audit found exact agreement between human and model scores for 194 of 200 revised samples.

Methodology in Plain English

The researchers first defined what a complete page parse should contain. A model must identify each element, locate it on the page, label it, place it in reading order, transcribe text when relevant, preserve table structure when applicable, and connect visual elements to captions and alternative text.

They built the dataset from open-access ACM scientific papers. Papers were split at the paper level into training, validation, and test sets using a 90:5:5 ratio. Annotators began with parser-generated regions and labels, then checked and corrected boxes, reading order, caption links, and other fields together. Visual descriptions were generated and scored with Gemini models, with human refinement for low-scoring cases.

For SARA, the authors froze the base vision-language model and reused visual features from intermediate encoder layers. The adapter aligns those features to the image-crop grid used by the model, then adds them to the model’s existing visual tokens through a learned residual connection. The model can then generate the same page-level schema as the other systems.

The experiments compared general-purpose vision-language models, document-focused models, and pipeline parsers. They assessed layout matching, overlap between predicted and reference boxes, OCR, reading order, and visual-description similarity.

Why This Matters

Research impact: Read-Parse-Describe makes it possible to test whether document systems produce a consistent page representation rather than succeeding at isolated subtasks. It also provides a way to study the connection between visual grounding, reading order, text extraction, and image description.

Potential applications:

• Accessible scientific papers: Generate structured page content and alternative text that could support screen readers and reflowed reading.
• Scholarly search and indexing: Organize paper text, figures, tables, captions, and page positions for retrieval.
• Document ingestion and mining: Convert scientific PDFs into structured records for downstream analysis.
• Extraction auditing: Inspect whether text, captions, and descriptions are attached to the correct visual regions.

Industry relevance: The work addresses a practical integration problem for document-processing systems: combining recognition, structure, and descriptions in one output. Its results suggest that adding spatial guidance may improve some structural capabilities without fine-tuning the whole base model. The paper does not establish that the generated accessibility descriptions are ready for deployment without further validation.

Future Directions

1. Broaden the document coverage. The dataset focuses on open-access ACM scientific PDFs and mainly the first eight pages, so future work could test other document sources and include more late-paper material such as appendices and references.
2. Validate descriptions further. The paper uses model-assisted annotation with human refinement; more evaluation could test description reliability and usefulness in real accessibility workflows.
3. Study performance across data scales and models. The paper reports that SARA’s structural advantage over full fine-tuning emerges at 50% and 100% of training data for Qwen3.5-2B, while results are weaker or mixed at smaller fractions. Further work could clarify when the adapter is most useful.
4. Improve evidence about generalization. The authors note that exposure of closed-source models to benchmark material during pre-training cannot be fully excluded; further evaluation could investigate this issue and test robustness beyond the benchmark’s source domain.

Target Audience

This paper is most relevant to researchers and practitioners working on document AI, vision-language models, OCR, layout analysis, reading-order prediction, scientific-paper processing, and accessibility technology. It should also interest teams building document ingestion or search systems who need to connect page structure, extracted text, and visual content in a single representation.

Authors’ abstract

Document understanding systems are typically evaluated as separate components via layout analysis, optical text recognition, reading-order prediction, and image captioning. This separation leaves scientific figures, captions, and visual semantics weakly connected, and fails to measure whether a model can produce a single page-level parse that is both structurally faithful and semantically accessible. We introduce Read-Parse-Describe, a unified dataset for document parsing with visual element description generation. Given a page image, models are required to output an ordered list of layout elements containing bounding boxes, semantic labels, reading-order indices, recognized textual content, and alternative text descriptions for visual elements. The benchmark is built from 2,402 documents, comprising 18,163 continuous pages, 125,218 layout elements, and 11,338 visual instances. Besides, we further propose a plug-and-play space-awareness enhancer that reuses multi-layer visual features and injects grid-aware spatial guidance into vision-language tokens for layout-aware generation. Experiments across end-to-end VLMs and pipeline parsers show that existing systems still struggle to jointly recover reading order and visual semantics, while our enhanced model achieves strong joint performance. These results highlight the need to move beyond isolated document parsing modules toward unified models that jointly read, parse, and describe complex documents. The new dataset and code will be made publicly available at https://github.com/yufanchen96/RPD