DKDS: A Benchmark Dataset of Degraded Kuzushiji Documents with Seals for Detection and Binarization

AI summary

Overview

Research area: Computer vision for cultural heritage — document image analysis, object detection, and document binarization applied to pre-modern Japanese Kuzushiji manuscripts.

Technical level: Intermediate. The paper is readable by anyone familiar with basic object detection and image-to-image models, but the binarization metrics (DRD, ASM) and GAN training details require some background.

Scope: The paper introduces DKDS, a new benchmark dataset of degraded Kuzushiji documents with synthetic seals, and reports baseline results for two tasks — Kuzushiji character and seal detection, and document binarization.

What This Paper Is About

Kuzushiji is a pre-modern Japanese cursive script that only a few thousand trained experts in Japan can still read, and OCR systems for it work reasonably well on clean documents but break down when documents are degraded or covered by seals. Seals are especially problematic because they overlap directly with handwritten characters, causing OCR systems to misread characters or to mistake seal inscriptions for body text. The authors build a public benchmark dataset that deliberately reproduces this interference — degraded Kuzushiji documents with randomly overlaid seals — and define two evaluation tracks on it.

Key Contributions

1. A new benchmark dataset. The authors introduce Degraded Kuzushiji Documents with Seals (DKDS), described as the first publicly available dataset specifically targeting the challenge of Kuzushiji characters overlapping with seals. It is built with the assistance of a trained Kuzushiji expert.

2. A documented construction pipeline including human-verified binarization ground truth. The dataset construction is described in detail, covering raw data collection, generation of binarization ground-truth via a pre-trained model, verification and manual correction by a Kuzushiji expert plus a second researcher, and the random placement of seal images onto the raw documents.

3. Definition of a Kuzushiji character and seal detection track with YOLO baselines. Track 1 is defined with evaluation via parameters, FLOPs, precision, recall and F-measure, and baseline results are provided for medium-sized YOLOv8, YOLOv9, YOLOv10 and YOLO11.

4. Definition of a document binarization track with traditional, GAN, and improved cGAN baselines. Track 2 baselines include traditional algorithms (Otsu, Niblack, Sauvola), three K-means clustering combinations, two state-of-the-art GAN methods, and the authors' own improved conditional GAN method.

Main Findings

• Dataset scale and composition. The training set contains 40 documents with 9,139 characters (228.5 per document) and 200 seals (100 seal images duplicated ×2, averaging 5.0 per document). Testing-E contains 10 documents, 2,145 characters (214.5 per document) and 28 seals (2.8 per document). Testing-D contains 10 documents, 2,145 characters (214.5 per document) and 140 seals (28 × 5, 14.0 per document). Testing-R contains 12 real documents with seals, 2,181 characters (181.8 per document) and 22 seals (1.8 per document).

• Source data. Raw images come from the book Genji Monogatari (The Tale of Genji), ID "200003803", provided by the Center for Open Data in the Humanities (CODH), at roughly 2100 × 3200 pixels, with OCR annotations covering 237 unique character types and 11,132 characters. The Testing-R subset uses 12 additional document images with seals from the CODH Kuzushiji dataset.

• Seal data. 219 seal images associated with three Qing dynasty emperors (Kangxi, 1661–1722; Yongzheng, 1722–1735; Qianlong, 1735–1796) were collected; 128 were selected as the final seal set (100 for training, 28 for testing). Seal images were resized randomly within 100–300 pixels and placed in a region anchored at (x=100, y=120) with width 1900 and height 2900 pixels, using RGBA alpha compositing rather than hard pixel replacement.

• Track 1, Testing-E. YOLO11 is best for Kuzushiji character detection with an F-measure of 94.8 (precision 97.7, recall 92.0). For seal detection, YOLOv9 is best with an F-measure of 98.1, followed by YOLOv10 at 98.0 and YOLOv8 at 95.9; YOLO11 is lowest at 94.9.

• Track 1, Testing-D. YOLO11 again leads Kuzushiji character detection with an F-measure of 91.9, and leads seal detection with an F-measure of 90.8, ahead of YOLOv8 at 86.8, YOLOv10 at 85.7 and YOLOv9 at 84.8. The authors attribute the drop in difficulty largely to decreased seal recall.

• Track 1, Testing-R. All models perform worse than on Testing-E and Testing-D. YOLOv9 is best for Kuzushiji characters (precision 91.6, recall 82.1, F-measure 86.6), and YOLOv8 is best for seals (precision 75.8, recall 85.7, F-measure 80.4).

• Track 1 model efficiency. YOLOv10 is smallest at 15.31M parameters and 58.9G FLOPs; YOLOv8 is largest at 25.84M parameters and 78.7G FLOPs; YOLOv9 is 20.01M / 76.5G and YOLO11 is 20.03M / 67.7G.

• Track 2, Testing-E. The authors' improved cGAN method achieves the best scores: FM 98.11, p-FM 98.14, PSNR 26.53dB, DRD 0.82, ASM 80.49. Ju et al. (KBS2024) reaches FM 95.80 / ASM 78.25, Suh et al. (PR2022) FM 93.09 / ASM 76.01, K-means + Sauvola FM 88.59 / ASM 73.09, Sauvola FM 87.87 / ASM 72.55, K-means + Otsu FM 84.76 / ASM 69.57, Otsu FM 63.01 / ASM 50.10, K-means + Niblack FM 39.99 / ASM 28.49, and Niblack FM 39.13 / ASM 27.25.

• Track 2, Testing-D. The improved cGAN again leads: FM 97.08, p-FM 97.13, PSNR 24.58dB, DRD 1.38, ASM 79.35, ahead of Ju et al. (FM 94.86 / ASM 77.40) and Suh et al. (FM 91.95 / ASM 75.04).

• Track 2, Testing-R. On real documents the ordering changes. Suh et al. (PR2022) performs best with FM 81.00, p-FM 81.53, PSNR 16.26dB, DRD 16.10, ASM 65.67, ahead of Ju et al. (FM 77.37 / ASM 59.96), while the authors' improved cGAN method scores FM 73.33, p-FM 72.98, PSNR 15.74dB, DRD 31.44, ASM 57.65. Sauvola reaches FM 73.69 / ASM 57.82, and Otsu FM 67.85 / ASM 49.36.

• Model size comparison for Track 2. The improved cGAN uses 31.22M parameters, versus 187.30M for Suh et al. and 191.46M for Ju et al. The traditional methods have no reported parameter count.

• Qualitative detection failures. On Testing-D, all YOLO models incorrectly detect stains in the upper-left corner as Kuzushiji characters, YOLOv8m repeatedly detects the same character in the bottom-left corner, and YOLO11m detects marginal notes that the ground truth excludes. On Testing-R, all models show varying degrees of duplicate seal detection.

• OCR evaluation (Section 8). Section 8 is described as presenting OCR evaluation results, but the provided paper content is truncated before that section, so those numbers are not reported here.

Methodology in Plain English

The authors start from clean, annotated Kuzushiji page images from Genji Monogatari and then manufacture a controlled, difficult version of them.

1. Collect raw pages. They take 50 document images from the CODH source, splitting 40 for training and 10 for testing.

2. Build binarization ground truth. They run a binarization model pre-trained on the DIBCO benchmarks over the raw pages to produce an initial black-and-white target. Because this initial output contained errors even without seals — missing marginal notes, voiced sound symbols and periods, and leftover stains — a trained Kuzushiji expert reviewed it, corrected it, and a second researcher re-verified the corrections.

3. Collect and prepare seals. They gather 219 high-resolution seal images from three Qing dynasty emperors and narrow this to 128, removing backgrounds so the seals are stored as PNG with transparency (RGBA).

4. Paste seals onto the pages. For each page, one or more seals are cropped to their bounding box, randomly resized to between 100 and 300 pixels, and placed at random positions inside a defined content region. Blending uses the alpha channel so the seal looks like it is partially covering the ink underneath rather than replacing it. Each pasted seal's bounding box is recorded as an annotation. To keep overlaps realistic, a new seal is re-positioned if it would overlap more than two existing seals.

5. Create difficulty tiers. Testing-E uses 28 seals (2.8 per document) and Testing-D uses those same 28 seals duplicated five times (140 seals, 14.0 per document). A separate Testing-R set uses 12 genuinely seal-bearing documents from CODH to test generalization to real material.

6. Run baselines. For detection, they train medium-sized YOLOv8, YOLOv9, YOLOv10 and YOLO11 for 100 epochs at 640 × 640 input with batch size 64, SGD and an initial learning rate of 0.01, initialized from COCO weights, on a single NVIDIA GeForce RTX 3090. For binarization, they compare traditional thresholding (Otsu, Niblack, Sauvola), versions with K-means clustering (K=3) applied first to strip reddish seal regions, two GAN methods, and their own cGAN. Their model splits each roughly 2100 × 3200 page into 512 × 512 patches, feeds them to a U-Net generator with an EfficientNet-B5 backbone and a PatchGAN discriminator, and trains with a combination of WGAN-GP loss and binary cross-entropy loss.

7. Score everything. Detection is scored with parameter count, FLOPs, precision, recall and F-measure. Binarization is scored with FM, pseudo-FM, PSNR, DRD and a composite Average-Score Metric (ASM) that averages FM, p-FM, PSNR and (100 − DRD).

Why This Matters

Impact on research. Existing Kuzushiji datasets such as Kuzushiji-MNIST, Kuzushiji-49 and Kuzushiji-Kanji, and existing binarization datasets such as the DIBCO series, the Bickley Diary and SMADI, either do not contain seals or are written in non-logographic scripts. The authors state that no publicly available document binarization dataset is specifically designed for pre-modern Japanese Kuzushiji documents, and that no degraded document dataset treats seals as a primary interference source. DKDS fills both gaps and provides reproducible baselines in one place.

Real-world applications:

• Digitization projects at libraries, archives and universities that need to convert scanned Kuzushiji holdings into machine-readable text.
• Preprocessing pipelines for Kuzushiji OCR, where binarization can restore characters hidden under seal ink before recognition (the paper's Figure 1 illustrates three concrete OCR failure modes that binarization addresses).
• Seal analysis in historical and provenance research, since seals carry collector names, personal wishes, interests or social status.
• Cultural heritage access tools, such as the "miwo" mobile application cited in the paper, which aim to let non-experts read pre-modern Japanese documents.

Industry relevance. The dataset is positioned directly against the practical bottleneck of noisy inputs rather than clean academic samples. The binarization baselines contrast a 31.22M-parameter improved cGAN against GAN baselines of 187.30M and 191.46M parameters, which is relevant to anyone deploying such preprocessing under memory or latency constraints. The detection baselines cover the current YOLO family, giving practitioners an immediate sense of the accuracy-versus-cost trade-off on this document type.

Future Directions

• Closing the real-document gap. The authors' improved cGAN leads on Testing-E and Testing-D but falls behind Suh et al. and Ju et al. on the real Testing-R subset. Understanding and fixing this synthetic-to-real drop is the most obvious next step, and the paper does not report an explanation for it.

• Improving seal recall under heavy interference. The paper identifies incomplete seal detection as the primary cause of the Testing-D performance drop. Reducing duplicate detections and partial detections, which the qualitative results show across all YOLO models, remains open.

• Reducing false positives from non-character elements. All YOLO models misdetect stains as Kuzushiji characters and detect marginal notes that the annotations exclude. Whether to include marginal notes as a labeled category, or to suppress them more aggressively, is unresolved.

• Broader seal and source coverage. The seal images come from 128 Qing dynasty imperial seals rather than ancient Japanese seals, which the authors attribute to the unavailability of high-resolution ancient Japanese seal images. Expanding to more authentic seal sources, layouts and calligraphic styles would test robustness further.

• OCR-level evaluation. The paper states that Section 8 presents OCR evaluation results to analyze the effectiveness of the dataset and methods, but that content is not present in the provided text, so those findings are not reported here.

Target Audience

This paper is most useful to researchers and engineers working on document image analysis for historical and non-Latin scripts — particularly those building OCR pipelines for Japanese manuscripts, evaluating detection models on degraded documents, or benchmarking binarization methods. It is also relevant to digital humanities teams, library and archive digitization groups, and machine learning practitioners who need a labeled dataset that pairs realistic degradation and seal interference with human-verified ground truth. Readers looking for a solved binarization method should note that the strongest results in the paper come with a clear caveat on real documents; readers looking for a ready-to-use dataset and baseline code will find them at the project page listed in the abstract.

Authors’ abstract

Kuzushiji, a pre-modern Japanese cursive script, can currently be read and understood by only a few thousand trained experts in Japan. With the rapid development of deep learning, researchers have begun applying Optical Character Recognition (OCR) techniques to transcribe Kuzushiji into modern Japanese. Although existing OCR methods perform well on clean pre-modern Japanese documents written in Kuzushiji, they often fail to consider various types of noise, such as document degradation and seals, which significantly affect recognition accuracy. To the best of our knowledge, no existing dataset specifically addresses these challenges. To address this gap, we introduce the Degraded Kuzushiji Documents with Seals (DKDS) dataset as a new benchmark for related tasks. We describe the dataset construction process, which involves the assistance of a trained Kuzushiji expert, and define two benchmark tracks: (1) Kuzushiji character and seal detection and (2) document binarization. For the Kuzushiji character and seal detection track, we provide baseline results using several recent versions of YOLO to detect Kuzushiji characters and seals. For the document binarization track, we present baseline results from traditional binarization algorithms, traditional algorithms combined with K-means clustering, two state-of-the-art (SOTA) generative adversarial network (GAN) methods, and our improved conditional GAN (cGAN)-based method. The DKDS dataset and the implementation code for baseline methods are available at https://ruiyangju.github.io/DKDS.