Beyond the Pipeline: Analyzing Key Factors in End-to-End Deep Learning for Historical Writer Identification

AI summary

Overview

• Research area: Computer vision / document analysis, specifically historical writer identification (HWI) using end-to-end deep learning pipelines.
• Technical level: Intermediate. Familiarity with convolutional networks, transformers, metric learning losses, and retrieval evaluation helps, but the paper's arguments are readable without deep mathematical background.
• Scope: A systematic study of how pre-processing, backbone architecture, loss function, and post-processing choices affect end-to-end deep learning for identifying writers of historical documents, evaluated on three ICDAR benchmark datasets.

What This Paper Is About

Historical writer identification is the task of deciding which person wrote a given handwritten historical document, based on stylistic patterns such as stroke curvature, letter formation, and spacing. The problem is hard because handwriting varies, documents are degraded, and there are few labelled examples per writer — sometimes making the task unreliable even for trained paleographers and forensic analysts. This paper asks which parts of an end-to-end deep learning pipeline actually matter, and whether a simple pipeline can match more complex state-of-the-art systems, especially in zero-shot settings where test writers never appear in training data.

Key Contributions

1. The authors explore multiple configurations of end-to-end deep learning pipelines for HWI to find setups balancing simplicity and performance.
2. They evaluate the impact of several pre-processing strategies — SIFT-based keypoint detection, document binarisation, and Text-AOI (area of interest) text region selection — on challenging historical document datasets.
3. They investigate how different backbone architectures, loss functions, and post-processing techniques affect performance, and compare results against more complex state-of-the-art approaches on widely used benchmarks.
4. Rather than proposing a single best model, they identify which design choices drive performance variation, particularly in zero-shot settings with unseen writers and diverse document layouts.

Main Findings

• Most end-to-end configurations perform poorly. The authors report that the majority of tested setups struggle to generalise in realistic document-level and zero-shot settings, attributing this to weak capture of low-level visual features, inconsistent patch representations, and high sensitivity to content noise.
• SIFT keypoints are unreliable on noisy historical scans. Because SIFT has no understanding of general text patterns, noisy artefacts are often selected as top candidates instead of actual text. With deep learning using larger patches than the SIFT keypoint neighbourhood, closely spaced keypoints produce highly similar input patches, reducing training diversity; on images with thousands of keypoints, random sampling from them behaves like naive random patch selection.
• Binarisation plus Text-AOI selection is the most effective pre-processing combination. The atrous binarisation model can still leave artefacts when scan quality differs from its training data, but adding Text-AOI selection filters out less relevant regions. Combined with random patch selection, this focuses the network on meaningful visual patterns.
• Transformer backbones outperform CNN backbones. On ICDAR2013-WI with Triplet loss and mean pooling, Swinv2-Small reached 31.50% Top-1 and 46.90% Top-5; Swinv2-Base 33.60% / 53.20%; Swinv2-Base + Cosine 39.80% / 58.10%; Swinv2-Large 40.20% / 61.10%. EfficientNetV2S reached 7.10% / 21.20% and ResNet18 17.40% / 33.50%. Swinv2-Large topped raw accuracy, but the authors note it is costly to train and to fine-tune further. The conclusion section instead states that SwinV2-Base with L2 regularisation achieved the highest accuracy.
• CNN loss curves show stagnation. Although ResNet18 had consistently lower training and validation loss, its validation loss was stagnant and followed a similar trend to EfficientNetV2, indicating the CNNs were not properly learning general pattern separation. SwinV2-Base showed steadier loss decline, with the cosine Triplet variant (using L2 regularisation) performing best.
• PCA dimension reduction generally does not help. On ICDAR2013-WI, SwinV2-Small fell from 31.50% (original) to 28.50% at PCA-128, PCA-64, and PCA-32; SwinV2-Base fell from 33.60% to 32.70%, 32.70%, and 32.00%; SwinV2-Large fell from 40.20% to 38.20%, 38.10%, and 37.60%. Only the L2-normalised SwinV2-Base Cosine model improved slightly, from 39.80% to 40.40% (PCA-128), 40.20% (PCA-64), and 39.20% (PCA-32).
• ArcFace fine-tuning is the single biggest win. Fine-tuning SwinV2-Base with ArcFace produced results comparable to more complex top-performing methods while keeping a simple pipeline. On ICDAR2013-WI, SwinV2 + ArcFace reached 97.40% Top-1 and 99.25% Top-5, versus 95.60% / 98.60% for Contour Gradient Rank-1, 98.70% Top-1 for NetVLAD + Re-Ranking, and 97% / 98.6% for ViT + KNN. On ICDAR2017-HistoricalWI it reached 97.15% Top-1, 96.08% P@2, and 42.16% mAP, compared with 76.4 / 68.4 / 55.6 for oBIFs columns histogram, 89.43 Top-1 and 78.20 mAP for Pair & Triple-SVM, and 88.9 / 84.8 / 76.2 for Clustering + CNN + Fisher + SVM.
• ArcFace beats Triplet loss for feature quality. The authors argue Triplet loss optimises only a limited number of triplets at a time, which is hard to tune in zero-shot settings with high handwriting variability, whereas ArcFace enforces angular margin constraints across all classes and yields more generalisable representations. The trade-off is slower, more resource-intensive training, making ArcFace better suited to fine-tuning than training from scratch.

Methodology in Plain English

The authors treat HWI as a pipeline that turns a document image into a compact feature vector, with three stages: pre-processing, feature extraction, and post-processing.

For pre-processing they compare three options. SIFT detects stable keypoints (corners and textured regions) and crops patches around them — a classic handcrafted approach that needs no training data. Historical document binarisation uses an atrous binarisation model with an atrous ResNet18 backbone, trained on DIBCO'09 to DIBCO'14 (including H-DIBCO handwriting data) with pseudo-F loss and validated on DIBCO'16, to produce a text mask. Text-AOI selection combines that binarisation with morphological dilation and connected component analysis: the binary mask is dilated to close small writing gaps, connected components are grouped into bounding boxes, boxes are ranked by area, and the top region is cropped from the original image as the area of interest. Dilation factors of (30, 30) were used for ICDAR2017-HistoricalWI and (25, 25) for ICDAR2019-HisIR, with manual adjustment for some miss-segmented images.

For feature extraction they compare SwinV2 transformers (Small, Base, Large) against ResNet18 and EfficientNetV2-S. Each backbone gets an MLP head projecting to a 128-dimensional feature vector (SwinV2 and EfficientNetV2S), 256-dimensional for SwinV2-Base trained with cosine distance, and no MLP for ResNet18 (to mimic a Siamese setup). The SwinV2 models are fine-tuned with LoRA on the query, key, and value parts of self-attention (r = 32, alpha = 64, dropout = 0.05), with pretrained weights frozen; the CNNs have early layers frozen. ResNet18 takes SIFT-centred crops as input while other models use random patches. Training used AdamW with a learning rate of 0.0001 and Triplet loss with margin = 1.0 and p = 2; a cosine-similarity Triplet variant with L2 regularisation was also tested; SwinV2-Base was then fine-tuned with an ArcFace head until loss convergence. The binarisation model used SGD with learning rate 0.001 and momentum 0.9.

For post-processing, patch features are aggregated by mean pooling across feature dimensions, followed by distance calculation using Euclidean distance or cosine similarity depending on the network setup. PCA on the feature vector before pooling is also tested at 128, 64, and 32 dimensions.

Datasets: ICDAR2013-WI (1400 images from 250 writers; 400 images from 100 writers for experiments and 1000 images from 250 writers for benchmarking; resized at r = 0.5, no pre-processing or Text-AOI needed). ICDAR2017-HistoricalWI (training set of 1182 images from 394 writers, test set of 3600 images from 720 writers, 4782 images from 1094 writers in total, with no writer appearing in both sets). ICDAR2019-HisIR (raw scanned historical documents for style-based retrieval, resized at r = 0.5 and r = 0.25 depending on scan size).

Why This Matters

Impact on research. The paper pushes back on the assumption that end-to-end deep learning automatically solves historical writer identification. By varying one design choice at a time, it shows that most configurations fail to generalise to unseen writers, and that the largest gains come from a specific combination: binarisation with Text-AOI selection in front of the model, a transformer backbone, and ArcFace fine-tuning rather than Triplet loss. It also offers a reproducible comparison point, showing a simple pipeline reaching 97.40% Top-1 on ICDAR2013-WI and 97.15% Top-1 on ICDAR2017-HistoricalWI, which is useful for researchers weighing complexity against accuracy. The paper notes that existing HWI datasets together cover only slightly above 10000 writers, with most of those in test sets with minimal annotation and uneven class representation — a data limitation that shapes what is achievable.

Real-world applications.

• Archival and library science: attributing unsigned or disputed manuscripts at scale, and clustering large unlabelled collections by scribe.
• Historical and literary scholarship: supporting authorship attribution and tracing document provenance across collections.
• Forensic document examination: supplementing subjective expert judgement with quantitative feature comparisons.
• Medical diagnostics: the introduction notes that handwriting analysis can help identify neurodegenerative diseases by tracking changes in a patient's writing over time.

Industry relevance. Document digitisation vendors and cultural-heritage digitisation programmes can use the pre-processing findings (binarisation plus Text-AOI selection) directly, since these steps work on raw, degraded scans rather than curated line crops. The ArcFace-versus-Triplet comparison is also relevant to anyone deploying retrieval or clustering systems for handwriting, where zero-shot generalisation to new writers matters more than fitting a fixed class list. The authors caution, however, that ArcFace's slower training makes it better suited to fine-tuning than to training large models from scratch.

Future Directions

• Developing more efficient fine-tuning strategies to reduce the cost of ArcFace-based training, which the authors identify as future work, especially as larger historical writer identification datasets become available.
• Revisiting feature-learnability failures: understanding why ResNet18 and EfficientNetV2S validation losses stagnated while transformer losses declined, and whether better training recipes exist for CNN backbones in HWI.
• Improving patch representation consistency, since inconsistent patch-level outputs and high sensitivity to content noise are identified as key obstacles; the SIFT-based sampling problem of generating near-duplicate patches is one specific target.
• Addressing the mAP result: on ICDAR2017-HistoricalWI the proposed SwinV2 + ArcFace system achieved 97.15% Top-1 and 96.08% P@2 but 42.16% mAP, below all three listed baselines (55.6, 78.20, and 76.2), leaving open how to reconcile strong rank-based retrieval with weaker mean average precision.
• Scaling evaluation to larger, more diverse and unevenly annotated historical collections, which the paper identifies as a limiting factor in the current literature.

Target Audience

Researchers and graduate students working on document analysis, writer identification, and historical document processing; practitioners building retrieval or clustering systems for digitised manuscripts; and computer vision engineers interested in how backbone choice, metric learning loss, and pre-processing interact in low-data, zero-shot retrieval settings. Digital humanities scholars and archivists evaluating whether automated attribution tools are viable for their collections may also benefit, but the paper is written for a technical audience comfortable with retrieval metrics such as Top-1, Top-5, P@2, and mAP.

Authors’ abstract

This paper investigates various factors that influence the performance of end-to-end deep learning approaches for historical writer identification (HWI), a task that remains challenging due to the diversity of handwriting styles, document degradation, and the limited number of labelled samples per writer. These conditions often make accurate recognition difficult, even for human experts. Traditional HWI methods typically rely on handcrafted image processing and clustering techniques, which tend to perform well on small and carefully curated datasets. In contrast, end-to-end pipelines aim to automate the process by learning features directly from document images. However, our experiments show that many of these models struggle to generalise in more realistic, document-level settings, especially under zero-shot scenarios where writers in the test set are not present in the training data. We explore different combinations of pre-processing methods, backbone architectures, and post-processing strategies, including text segmentation, patch sampling, and feature aggregation. The results suggest that most configurations perform poorly due to weak capture of low-level visual features, inconsistent patch representations, and high sensitivity to content noise. Still, we identify one end-to-end setup that achieves results comparable to the top-performing system, despite using a simpler design. These findings point to key challenges in building robust end-to-end systems and offer insight into design choices that improve performance in historical document writer identification.