Barnamala: Parameter-Efficient Handwritten Devanagari Recognition at Benchmark Saturation

AI summary

Overview

• Research area: Computer vision / handwritten character recognition, specifically Indic script (Devanagari) OCR, with a focus on statistical evaluation methodology and benchmark saturation.
• Technical level: Intermediate. The architecture is a standard compact convolutional network, but the paper's central argument depends on familiarity with knowledge distillation, exact McNemar testing, and Wilson confidence intervals.
• Scope: The paper presents a 1.11 M-parameter convolutional student model for 46-class DHCD Devanagari recognition, argues that the benchmark is saturated at a shared 11-error floor, and reports transfer and corruption-robustness results against a reproduced 17.32 M-parameter baseline.

What This Paper Is About

DHCD is the standard 46-class handwritten Devanagari benchmark, and top-1 accuracies have crowded above 99.7 percent, with successive gains measured in hundredths of a percent. The authors argue that these rankings carry little inferential weight at this level and that paired statistical testing has not been applied rigorously. Their goal is to build a much smaller model that reaches statistical parity with the largest prior state of the art, and then to demonstrate that the benchmark itself, not model capacity, now limits progress.

Key Contributions

1. A compact model at parity with the state of the art. Barnamala carries 1.11 M parameters and reaches 99.73 percent on the standard 46-class DHCD split, reported as the highest accuracy on that split at 15.6 times smaller footprint than prior state of the art. An exact McNemar test against the reproduced baseline gives p = 0.345, confirming parity even without knowledge distillation.

2. Evidence that DHCD is saturated. Exact McNemar tests and Wilson confidence intervals applied across all configurations reveal a shared 11-error intrinsic floor. Both the 15 student models and the reproduced baseline fail on the same 11 images, and no configuration classifies any of them correctly regardless of architecture, training strategy, or random seed.

3. Ablations that close off the obvious escapes. Distillation target cleanliness (TTA-softened versus clean), ensemble size (9 versus 15 teachers), and test-time augmentation each produce no statistically significant gain once noise is accounted for. More teachers do not move the ceiling: both the 9-teacher and 15-teacher clean majorities plateau at 30 errors with an identical McNemar p = 0.0755.

4. Transfer and robustness at reduced cost. At 15.6 times smaller and 9.5 times faster on CPU than prior state of the art, the model transfers to CMATERdb digits (zero-shot 76.6 percent; fine-tuned 97.8 percent) and degrades far less under corruptions (mean corruption accuracy 75.7 percent versus 38.7 percent). Full reproduction materials are at https://github.com/Ampixa/barnamala.

Main Findings

• Parameter efficiency without accuracy loss: The distilled, TTA-targets variant reaches 36.6 ± 2.1 errors over 5 seeds (99.735 percent) with McNemar p = 0.345 against the reproduced 17.32 M-parameter baseline. The best seed reaches 34 errors (99.754 percent), with Wilson 95 percent CI [99.656 percent, 99.824 percent].

• Distillation is not the source of parity: The clean single-pass logits variant gives p = 1.000, and the supervised control trained with no teacher signal at all ties at p = 0.701 (39.8 ± 2.8 errors, 99.712 percent). Only three errors separate the best distilled student from the supervised baseline.

• Target recipe barely matters: Distilled (TTA) targets yield 36.6 ± 2.1 errors; distilled (clean) targets from a larger 15-teacher pool yield 38.2 ± 2.2 errors. The two are statistically indistinguishable.

• Ensemble size does not move the ceiling: Both the 9-teacher and 15-teacher clean majority pools plateau at 30 errors (99.783 percent) with identical McNemar p = 0.0755. Individual clean teacher errors range from 27 to 43, so the pools have diversity, but the residual errors sit in an irreducible floor where additional models vote the same wrong answer.

• The one sub-0.05 result is rejected on methodological grounds: Flip-TTA at test time gives 28 errors (99.797 percent, p = 0.029), but horizontal reflection of Devanagari produces characters that do not exist in the script, and the gain does not survive when applied to individual teachers, where errors rise to 88 and 102 at the most affected checkpoints.

• A hard significance frontier: Beating the reproduced baseline (40 errors) under an exact two-sided McNemar test at alpha = 0.05 requires at most 25 errors (at least 99.82 percent accuracy). A comfortable margin means at most 20 errors (at least 99.855 percent), which would demand resolving nearly every ambiguous glyph pair.

• Errors outside the floor are model-specific: Average pairwise Jaccard similarity across models is 0.48, meaning seeds shuffle which images fail, but ensembles and seeds reduce variance rather than the bias imposed by the shared floor.

• Top confusion pairs at 32x32: ba/waw (17 cross-confusions), waw/tabala (16), tra/ba (15), dha/gha (13), and da/dhaa (9). At this resolution the distinguishing strokes span only a few pixels, and DHCD provides neither high-resolution input nor stroke-level annotations.

• Calibration does not covary with accuracy: Expected Calibration Error falls in the range 0.13–0.16 across all Barnamala configurations, and the 11 floor images are confidently wrong rather than uncertain at every seed.

• Efficiency gains are large and unassisted: 164 M MACs versus 2434 M, 4.4 MB versus 69.3 MB model size, 2.8 MB versus 26.1 MB peak activation, and 8.28 ± 0.20 ms versus 78.8 ± 0.89 ms CPU latency (151 img/s versus 13.6 img/s). Ratios are 15.6x, 14.8x, 15.6x, 9.5x, 9.5x, and 11.1x respectively, with no quantization or pruning.

• Digit transfer favors the small model: Zero-shot on CMATERdb, the distilled student reaches 76.6 ± 3.7 percent versus 62.7 percent for a supervised baseline trained on DHCD digit images only, a gap of 14 percentage points under domain shift despite being tied in-distribution. A linear probe reaches 85.4 percent, and 30 epochs of fine-tuning at a low learning rate reach 97.8 percent.

• Corruption robustness is uneven: Overall mCA is 75.7 percent versus 38.7 percent, with especially large gaps for blur (97.0 percent versus 37.6 percent) and contrast (75.5 percent versus 11.5 percent). The ranking reverses for additive noise, where the baseline is better (66.9 percent versus 54.7 percent).

• Cross-dataset generalization gaps are large: On NHCD (Pant 2012, 46 classes, n = 10,260), Barnamala reaches 78.92 percent zero-shot (72.33 percent consonants, 95.80 percent digits) while the baseline collapses to 23.99 percent (10.38 percent consonants, 58.85 percent digits). On the Prashanth et al. 2021 digit dataset (22,500 images), Barnamala reaches 79.31 percent versus 72.60 percent.

• A shared failure mode on cross-dataset tests: The dental/retroflex pairs (ta/ṭa, tha/ṭha, da/ḍa, dha/ḍha) collapse to near-zero accuracy for both models, caused by interclass confusion between strokes rendered differently in Indian and Nepali handwriting traditions.

Methodology in Plain English

The authors designed a small pre-activation squeeze-and-excitation residual network for single-channel 32x32 grayscale images. Channel widths run (40, 80, 160) across three stages of depth (2, 2, 2), with stages 2 and 3 downsampling by stride 2, and a squeeze-and-excitation ratio of 8. The total trainable parameter count is 1,109,116.

They trained a 15-teacher ensemble drawn from three architectural configurations crossed with five random seeds (0 to 4). Two configurations share widths (96, 192, 384) and depths (3, 3, 3) but differ in augmentation intensity; the third uses widths (64, 128, 256) with depths (4, 4, 4). Per-teacher parameter counts are 5.97 M and 9.89 M. The spread across configurations is intentional to discourage mode collapse in the soft targets. All teachers use the same Barnamala codebase and are trained without distillation, and their logits are dumped once into a single NumPy archive reused across all student runs.

The distillation objective combines a temperature-softened KL divergence with a hard cross-entropy term, using temperature T = 4.0, alpha = 0.7, and label smoothing of 0.1. The primary ensemble target is the mean softmax of nine teacher checkpoints, each computed with horizontal-flip averaging over the training image. Critically, because Devanagari is not horizontally symmetric, horizontal flips are completely excluded from training augmentation; instead the pipeline applies affine perturbations followed by elastic deformations at a chosen tier (light, medium, or heavy), plus random erasing (p = 0.25, area 2–10 percent). Mixup (alpha = 0.2) and CutMix (alpha = 1.0) are selected with equal probability and each activated for a batch with overall probability 0.5, and are turned off during distillation. Optimization uses AdamW with a cosine schedule preceded by a 5-epoch linear warm-up, and an EMA shadow copy with decay 0.999 supplies the validation and final checkpoint weights.

To make sample-level comparisons possible, the authors reimplemented the prior state-of-the-art architecture from its published description, including exact normalization (mean = 0.5, std = 0.5) and the class-index permutation implied by the data loader. Seed 0 produces 40 errors (99.7101 percent), matching the reported number. Evaluation uses a single forward pass over the held-out test split with no test-time augmentation, model selection on a 10 percent stratified validation set carved from the training split before augmentation, exact two-sided McNemar tests on per-sample binary correctness indicators, Wilson score intervals at 95 percent, and ECE computed with 15 equal-width bins.

Why This Matters

Impact on research. The paper reframes a leaderboard problem as a measurement problem. Its argument is that raw accuracy rankings on DHCD can flip between runs with no change in model quality once the best models err on fewer than 0.3 percent of examples, and that a gain of a few hundredths of a percent is meaningless without paired evaluation such as McNemar or an equivalent test. It also shows that a parameter reduction of 15.6 times costs nothing in accuracy at this benchmark's saturation point, which undercuts the assumption that larger models are the path forward here.

Real-world applications.

• On-device Devanagari handwriting recognition where memory and CPU budgets are tight, given the 4.4 MB model size and 8.28 ± 0.20 ms CPU latency.
• Digit-only form and document processing in Devanagari, given the 97.8 percent fine-tuned CMATERdb result and 85.4 percent linear-probe result.
• Deployment in blur-prone or low-contrast scanning conditions, where the model's mCA is 97.0 percent and 75.5 percent respectively against 37.6 percent and 11.5 percent for the baseline.
• Applications that must survive writer and scanner shifts across separate collection pipelines, given the 78.92 percent zero-shot NHCD result versus 23.99 percent for the baseline.

Industry relevance. The practical message is that compactness is the right operating point once remaining errors are governed by data rather than model capacity, and that a larger network has little room to justify its cost. The efficiency table gives concrete deployment numbers, and the paper's comparison against a 17.32 M-parameter baseline at 78.8 ± 0.89 ms CPU latency shows the tradeoff quantitatively rather than rhetorically.

Future Directions

• Build harder benchmarks. The authors state the saturation claim is tied to DHCD's particular conditions: its 32x32 resolution, concentrated label noise, and writer pool. They call for high-resolution scanning or a completely independent pool of writers, and note that cleaner or higher-resolution benchmarks may expose capacity differences DHCD can no longer reveal.

• Extend transfer evaluation to consonants. The transfer experiment covers only the digit subset (10 of 46 DHCD classes), a choice the authors attribute to available data rather than design preference. Doing this would require a genuinely independent target set sharing DHCD's label taxonomy, since the apparent public candidates are same-provenance splits of DHCD itself.

• Resolve the ambiguous glyph pairs. Any model hoping for a comfortable margin would need to resolve nearly every pair in the confusion table, such as ba/waw, waw/tabala, and tra/ba, which may require higher resolution or stroke-level annotation that DHCD does not provide.

• Address the noise-robustness reversal and the cross-dataset domain gap. The baseline beats Barnamala on additive noise, which matters for noisy deployment environments, and the dental/retroflex collapse on NHCD is a character-level domain gap that neither model resolves. Both remain open.

Target Audience

Researchers working on Indic script recognition and handwritten character recognition who need to interpret near-ceiling leaderboard results, and practitioners selecting models for resource-constrained Devanagari deployment. It is also directly relevant to anyone designing evaluation protocols for saturated benchmarks, since the McNemar and Wilson-interval methodology, the confusable-pair analysis, and the shared 11-error floor argument are the paper's distinctive contributions. Readers should be comfortable with model compression, knowledge distillation, and basic hypothesis testing concepts.

Authors’ abstract

We built a compact convolutional network (1.11 M parameters) for 46-class DHCD Devanagari recognition and reached 99.73%, the highest reported at 15.6x smaller than prior state-of-the-art. We have effectively reached the saturation point: every model tested, large teacher ensembles included, hits the same 11-error intrinsic floor. No configuration achieves a statistically clear win under exact McNemar tests with Wilson confidence intervals. Even without knowledge distillation, our student matches the nearest large-model baseline (17.32 M parameters; McNemar $p = 0.345$). Outside of DHCD, zero-shot on CMATERdb digits gives 76.6% and fine-tuning reaches 97.8%; corruption robustness is also far better than large baselines (mean corruption accuracy 75.7% vs. 38.7%). All artifacts are at https://github.com/Ampixa/barnamala.