Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts

AI summary

Overview

Research area: Handwritten Text Recognition (HTR) for historical, low-resource scripts — specifically Old Nepali written in Devanagari — combining transformer-based vision-language models with data-centric training strategies.

Technical level: Intermediate. The paper assumes familiarity with encoder-decoder architectures, tokenization, and error metrics (CER), though each component is explained clearly enough for a reader with general machine learning background.

Scope: The paper builds and evaluates an end-to-end three-stage HTR pipeline for Old Nepali manuscripts, achieving a best Character Error Rate (CER) of 4.9% and releasing training code, model configurations, and evaluation scripts.

What This Paper Is About

Historical Nepali manuscripts from the 18th and 19th centuries are written in Devanagari script in continuous writing without consistent word spacing, and they survive in variable handwriting, degraded scan quality, and complex conjunct character forms. Very little labeled handwritten Devanagari data exists, so standard OCR tools fail badly on them. The authors set out to build the first end-to-end HTR pipeline for Old Nepali by combining transfer learning from synthetic and printed data, data augmentation, label normalization, and script-aware tokenizers and decoders.

Key Contributions

1. A complete HTR pipeline for Old Nepali, reported as the first end-to-end pipeline for the language, evaluated against both a fine-tuned TrOCR baseline and Google Cloud Vision OCR.
2. A three-stage domain adaptation scheme that pretrains on synthetic Devanagari images, transfers to printed Nagari scans from heiDATA, and finally fine-tunes on 3,100 line-level images extracted from 155 Old Nepali manuscripts.
3. A systematic study of encoder, decoder, and tokenizer combinations (12 configurations across 3 encoders, 2 decoders, and 2 tokenizers, run through 3 training stages for 36 total runs), plus an investigation of five decoding strategies.
4. Detailed preprocessing, augmentation, and error analyses, including a character-level confusion analysis and an uncertainty-based flagging method, with code, model configurations, and evaluation scripts released publicly.

Main Findings

• Best result: 4.9% CER. The final model, using a trocr-large-handwritten encoder with a script-aware decoder, reached CER 0.049, weighted CER 0.048, and exact-match accuracy (ACC) of 33.5%.
• 49% relative CER reduction over the baseline. A fine-tuned full TrOCR baseline achieved CER 0.096, which the authors' pipeline reduced to 4.9%.
• Data-centric steps outperformed architectural tuning. Label normalization alone lowered CER from 0.089 to 0.084; adding augmentation at 8× brought it to 0.056. Going to 12× and 16× did not improve further, indicating a plateau.
• Decoder architecture and tokenizer mattered little. Across the 12 configurations, the best was a BERT decoder with byte-level BPE (CER 0.082 after all three stages), but the spread across decoders and tokenizers was small.
• Larger encoder helped. Upgrading from trocr-base-handwritten (CER 0.056, ACC 29.4%) to trocr-large-handwritten (CER 0.049, ACC 33.5%) was one of the clearer architectural gains.
• Swin Transformer underperformed. The swin-base-patch4-window7-224-in22k encoder reached CER 0.174 and ACC 21.93%, versus 0.056 and 29.35% for the TrOCR encoder under identical conditions.
• Binarization hurt slightly. Binarized images produced CER 0.098 versus 0.089 for non-binarized images, so non-binarized images were used.
• Decoding strategy had minimal effect. All beam sizes tested (1, 5, 10, 20) gave an identical weighted CER of 0.0483; contrastive search and sampling methods gave CER approximately 0.0488.
• Efficiency advantage over the standard decoder. The script-aware decoder achieved CER 0.056 with 202M parameters at 4.96 samples/sec, versus the TrOCR decoder's 0.059 with 334M parameters at 3.50 samples/sec — described as 1.4× faster evaluation.
• Errors are structured, not random. Confusions such as y (ya) with p (pa) and t (ta) with n (na) follow visual similarity, and a subset of 10 characters out of 80 accounted for 55.9% of all errors.
• Long lines are the main weakness. Errors rise sharply above 120 characters; only 26 of 3,100 lines exceed that length. Splitting such lines into two parts reduced errors in one case from 23 to 4 and in another from 30 to 2.
• Uncertainty can flag errors. Comparing relative probabilities of top predictions flagged about 27% of all errors, and over half of those could be recovered from the top-3 predictions.
• Google Cloud Vision OCR failed on this data, missing conjuncts, diacritics, and punctuation.

Methodology in Plain English

The authors treat recognition as a line-level problem. Using Kraken's polygon-based segmentation, they cut 155 manuscript images into 3,100 individual text lines (average line image size 1593 × 133 pixels; the full pages average 3091 × 3487 pixels at 328 dpi).

Training happens in three stages. First, a model is trained on 105,000 synthetic line images generated from historical Nepali textbooks, rendered in 11 Devanagari fonts with 10 types of noise and distortion applied at random. Second, the model is trained on 5,139 line images of printed Devanagari scanned from the heiDATA repository, converted to grayscale to resemble the manuscript data. Third, it is fine-tuned on the 3,100 real manuscript lines, split 80/10/10.

Architecturally, the system pairs a pretrained TrOCR vision encoder (either base or large) with a text decoder that is either BERT-based or GPT-2-based, trained from scratch with a custom Devanagari tokenizer using either character-level or byte-level BPE with a vocabulary of 500. All stages use AdamW, a learning rate of 3e-5, batch size 8, weight decay 0.01, 500 warmup steps, and seed 42, with 6, 10, and 20 epochs respectively.

To improve results, the authors normalize transcriptions (for instance, unifying inconsistent representations of the Devanagari chandrabindu, U+0310 and U+0901) and apply 20 augmentation variations at 2×, 4×, 8×, 12×, and 16× intensity, grouped into shape/angle distortions, quality degradations, and character-level distortions. Evaluation uses CER, length-weighted CER, and exact-match accuracy, with zero-width Unicode characters (U+200B, U+200C, U+200D) stripped from both predictions and references.

Why This Matters

Impact on research: The paper argues that in very low-resource historical OCR settings, curating data — cleaning labels and augmenting images — can matter more than scaling model architecture. It also provides a reproducible pipeline and a strong baseline for a script family that has received little HTR attention in machine learning venues.

Real-world applications:

• Digitizing and searching manuscript archives held by institutions such as the National Archives in Kathmandu and the Nepal German Manuscript Preservation Project collections.
• Assisting historians and philologists who currently transcribe royal edicts, petitions, and chronicles by hand.
• Building human-in-the-loop transcription tools, where the model's uncertainty scores direct reviewers to the lines most likely to need correction.
• Enabling downstream text analysis — linguistic, historical, or legal — on Old Nepali sources once they are machine-readable.

Industry relevance: The work is relevant to cultural heritage technology, digital library and archival platforms, and document AI vendors whose general-purpose OCR systems fail on non-Latin historical scripts. The paper's efficiency comparison (fewer parameters, faster evaluation) also speaks to deployment constraints in resource-limited archival settings.

Future Directions

• Expand the corpus. The authors call for collaboration with digital archives to address the scarcity of long-sequence training examples, which is the identified cause of degraded performance beyond 120 characters.
• Post-correction using structured errors. Because the model's mistakes follow interpretable visual patterns, correction methods could be built on them — though the absence of explicit word boundaries in the data makes this harder.
• Better segmentation for irregular layouts. The current pipeline depends on Kraken for pre-segmented line inputs, which the authors flag as an error source for damaged or unusually laid-out documents.
• Test generalization to other scripts and collections. The paper states that applying the approach to other historical manuscripts would validate its generalizability, and that further validation on additional collections is needed. It also notes that optimization parameters such as warmup steps remain unexplored future work.

Target Audience

This paper is most useful to researchers and engineers working on handwritten text recognition and low-resource OCR, especially those dealing with non-Latin or historical scripts. It is also valuable to digital humanities scholars, archivists, and librarians involved in manuscript digitization, and to machine learning practitioners interested in transfer learning, augmentation, and tokenization strategies for small datasets. Readers need some familiarity with transformer architectures and character error rate metrics to follow the experimental details, though the high-level pipeline and findings are accessible to a broader technical audience.

Authors’ abstract

This paper presents the first end-to-end pipeline for Handwritten Text Recognition (HTR) for Old Nepali, a historically significant but low-resource language. We adopt a line-level transcription approach and systematically explore encoder-decoder architectures and data-centric techniques to improve recognition accuracy. Our best model achieves a Character Error Rate (CER) of 4.9\%. In addition, we implement and evaluate decoding strategies and analyze token-level confusions to better understand model behavior and error patterns. Although the evaluation dataset is confidential, we release our training code, model configurations, and evaluation scripts to support further research on HTR for low-resource historical scripts.