Towards Hierarchical Structure Understanding of Newspaper Images

AI summary

Overview

Research area: Computer vision and document understanding, specifically hierarchical structure extraction from historical newspaper page images.

Technical level: Intermediate. The paper is readable without deep expertise, but understanding the two architectures assumes familiarity with object detection (YOLO), transformers, Swin encoders, and evaluation metrics such as mAP, IoU, BLEU, and Jaccard indices.

Scope: The paper compares two systems for recovering the nested section / article / block structure of historical newspaper pages, and releases a new annotated dataset (Finlam La Liberté) with an accompanying evaluation framework.

What This Paper Is About

Newspaper pages are dense and deeply nested: a page contains sections, sections contain articles, and articles contain content blocks such as titles, paragraphs, illustrations, captions, and tables, all arranged in a non-linear reading order. Existing methods either solve isolated sub-problems or rely on handcrafted rule-based pipelines that do not generalize well. The goal of this paper is to reconstruct that full hierarchy automatically from page images, and to compare a modular bottom-up pipeline against a single end-to-end transformer model on the same newly released dataset.

Key Contributions

1. Tiramisu (Tiered Transformers for Hierarchical Structure Understanding): an end-to-end transformer-based architecture that models document hierarchy through an iterative, tiered decoding process, performing section and article separation, block localization, semantic categorization, and reading order prediction with a Swin encoder and a transformer decoder with specialized heads.
2. A modular bottom-up pipeline combining open-source models — YOLO26 for block detection and classification, OpenCV Line Segment Detection (LSD) for separators, LayoutReader for reading order, and a custom article/section segmentation algorithm.
3. The Finlam La Liberté dataset: 1500 complete issues of the French-language newspaper La Liberté from the 19th and 20th centuries, curated by the French National Library, with page-level semantic zones, bounding boxes, OCR text, multi-level classification, and article/section separation.
4. A comprehensive evaluation framework with metrics for block detection, article segmentation, reading order, and structural counts, plus a head-to-head comparison of the two proposed systems against the commercial Arcanum service.

Main Findings

• Block detection and classification (Table 1): The bottom-up pipeline achieved the highest scores, with mAP@50 of 72.27% and mAP@50-95 of 59.08%. Tiramisu reached 39.14% mAP@50 and 22.05% mAP@50-95. Arcanum, evaluated zero-shot, scored 11.12% and 6.48%.
• Cascading misses in Tiramisu: The paper attributes Tiramisu's lower block scores to its hierarchical passes — if an article or section is missed in an early pass, all of its constituent child blocks are missed as well.
• Article segmentation (Table 2, AS@0.5): The pipeline led on surface-based metrics with an F1 of 80.39% and mIOU of 88.82%. Tiramisu led on count-based metrics with a Jaccard Article of 89.00% and Jaccard Section of 83.78%. Arcanum obtained an F1 of 59.42%, mIOU of 82.37%, Jaccard Article of 57.28%, and Jaccard Section of 50.22%.
• Reading order (Table 3): The pipeline achieved the best block-level BLEU of 87.20%, while Tiramisu scored 73.31% and Arcanum 55.58%. At the article level, both proposed methods were comparable and high: Tiramisu 97.43%, pipeline 97.08%, Arcanum 81.32%.
• Inference speed (Table 4): The pipeline ran at 0.83 seconds per image on an A100 GPU and 1.66 seconds on an i5-12450H CPU. Tiramisu required 1.46 seconds on GPU and 57.03 seconds on CPU. The pipeline is therefore x34 faster on CPU and x1.75 faster on GPU.
• Diagram-level mismatch, not failed detection, explains Arcanum's low strict scores: The paper notes that Arcanum tends to produce larger zones while the other methods create a new zone at each indented line, and that mapping Arcanum's native classes onto the paper's detailed schema introduces classification mismatches that penalize strict overlap metrics. Qualitatively, the authors state Arcanum's segmentations look acceptable.
• Trade-off between accuracy and deployment simplicity: Tiramisu is a single unified model that is straightforward to deploy and update, whereas the pipeline introduces a dependency chain across four stages (a dependency chain of three models is mentioned in Section 7, and the conclusion refers to four stages).
• Tiramisu training depends on synthetic data: The generator compensates for incomplete or noisy annotations and adds layout and structural diversity; the real-data proportion starts at 1% and increases by 1% per epoch up to a cap of about 40%.
• Dataset limitations: OCR quality does not match current state-of-the-art models, and while segmentation and reading order are mostly reliable, specific pages contain errors. A gold-standard dataset is in development.

Methodology in Plain English

The researchers define a formal, simplified representation of a newspaper as a Title Section (a sequence of blocks) plus a Content Section (a sequence of sections), where each section is a sequence of articles and each article is a class label plus a HEAD sequence and a BODY sequence of blocks. Each block carries text tokens, a class label from a context-specific label set (head, body, or title-section), and spatial coordinates (x, y, w, h). Both systems target this same representation.

The bottom-up pipeline chains specialized components. YOLO26 detects and classifies unit blocks; it was trained on images resized to 1024 pixels, batch size 8, for 250 epochs with early stopping and patience of 100, with the best performance at epoch 116 after 15 hours of training. Advertisement and free-ad zones were excluded from training. At inference, detections below 0.2 confidence are discarded and overlapping blocks are removed by Non-Maximal Suppression. LayoutReader then predicts reading order from normalized bounding box coordinates (0 to 1000) and predicted class integers, aided by vertical and horizontal separators found with OpenCV's Line Segment Detection; it was fine-tuned from the hantian/layoutreader checkpoint for 30 epochs with batch size 8 and learning rate 5e-5, and boxes are pre-sorted left-to-right and top-to-bottom at inference. Finally, a custom hierarchy updater walks regions in predicted reading order and starts a new article or section whenever an ARTICLE-TITLE or SECTION-TITLE zone (or group of zones) appears.

Tiramisu uses a Swin transformer encoder to turn the page image into image tokens, plus a transformer decoder that also receives structured elements from the previous hierarchy level. The decoder branches into specialized heads: article classification, block classification, block coordinate regression, and token prediction. Decoding proceeds in four passes — Pass 1 extracts sections (with the first block and first token of each), Pass 2 identifies articles within each section, Pass 3 retrieves all blocks within each article, and Pass 4 extracts the token content of each block. Special prompt tokens <lvl1> through <lvl4> (and matching closing tokens) signal the current pass, and prompts for pass k are built from the outputs of pass k-1 — for example, pass 2 is applied once per section predicted in pass 1. Embedding dimensions used are d = 1024, d_token = 1024, d_class_art = 8, d_class_block = 16, and d_coords = 8. Classification heads use cross-entropy loss; bounding boxes use a combination of Generalized Intersection over Union and Mean Squared Error. Because the available real annotations are noisy and limited in variety, a synthetic newspaper generator supplies training pages on the fly in an asynchronous workflow, using French Wikipedia content plus synthetic images and advertisements, and simulating printing and scanning artifacts such as blur, transparency, smudges, and stains.

Evaluation covers block detection with mAP@50 and mAP@50-95, article separation with F1 and mIoU under Hungarian-algorithm matching plus Count Jaccard indices for articles and sections, and reading order with BLEU computed on matched element indices at the section level and on class labels at the block level.

Why This Matters

This work addresses a bottleneck in large-scale digitization of historical press: recovering not just text but the logical hierarchy (which text belongs to which article, in which section, in what reading order) that makes archives searchable and navigable. It also provides a rare direct comparison between a fully modular system and a fully end-to-end system on the same task, dataset, and metrics.

Real-world applications:

• Digital humanities and historical research: enabling scholars to search, filter, and analyze newspaper content by article and section rather than as raw page images or unstructured OCR text.
• National library digitization programs: the paper states that by the end of 2026 the French National Library will implement the bottom-up pipeline and test it on its newspapers collection of over 8 million documents.
• Media analysis and press history: supporting large-scale studies of advertising formats, layout evolution, and editorial structure across decades.
• Information retrieval with spatial grounding: because both systems predict bounding boxes, end users can locate retrieved information precisely on the page, and curators can perform quality control on digitization outputs.

Industry relevance: commercial services exist in this space — Arcanum is described as having processed over 100 million pages across multiple European archives — but its pipeline is proprietary and operates as a black box. This paper offers open, reproducible alternatives and open tooling (training code, synthetic generator, dataset, and evaluation framework) that organizations can inspect, adapt, or plug different models into.

Future Directions

• A gold-standard dataset: the paper states a gold-standard dataset is in development to address annotation errors and OCR quality that lags behind current state-of-the-art models in Finlam La Liberté.
• Reducing cascading errors in Tiramisu: early-pass misses propagate to all child blocks; improving early-pass recall or adding recovery mechanisms is an open problem.
• Scaling to full library collections: the announced deployment of the bottom-up pipeline at the French National Library on over 8 million documents will test whether these methods hold up at production scale.
• Balancing end-to-end simplicity against modular accuracy: Tiramisu is easier to deploy as a single model, while the pipeline is more accurate and faster but harder to maintain — how to close that gap, and how far synthetic data can substitute for clean annotations (currently capped at roughly 40% real data in Tiramisu training), are open questions.

Target Audience

Researchers and engineers working on document understanding, layout analysis, and historical document digitization; digital humanities and library technology teams evaluating automation for newspaper collections; and machine learning practitioners interested in hierarchical, structure-aware transformer architectures or in the practical trade-offs between modular pipelines and end-to-end models.

Authors’ abstract

Understanding newspaper images remains a challenging task due to their complex, nested hierarchical structures and dense, heterogeneous layouts. In this paper, we explore two complementary approaches for newspaper structure understanding. First, we present a modular bottom-up pipeline that combines state-of-the-art open-source models: YOLO for layout detection, LayoutReader for reading order prediction, and a custom algorithm for article segmentation. This approach leverages existing robust components while maintaining flexibility and interpretability. Second, we introduce Tiramisu (Tiered Transformers for Hierarchical Structure Understanding), a novel end-to-end transformer-based architecture that explicitly models document hierarchy through an iterative tiered process. Tiramisu performs section and article separation, block localization, semantic categorization, and reading order prediction using highly parallelized attention mechanisms. Finally, we release Finlam La Liberté, a new dataset designed specifically for evaluating hierarchical information retrieval in historical newspapers. Experimental results demonstrate the effectiveness of both approaches in reconstructing complex newspaper hierarchies, with comparative analysis highlighting their respective strengths for scalable document digitization. The Tiramisu training code, including the synthetic newspaper generator, is available at https://git.litislab.fr/tiramisu/tiramisu-newspaper-articles-extractor.