Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

AI summary

Overview

Research area: OCR and vision–language model (VLM) evaluation metrics; NLP evaluation methodology.

Technical level: Intermediate.

Scope: The paper introduces semCER, a variant of Character Error Rate that keeps CER's alignment and denominator but replaces each edit's unit charge with an LLM-judged severity, and validates it against human readers in three languages and across a 16-language, 61-million-edit corpus.

What This Paper Is About

Character Error Rate (CER) is the standard metric for OCR and VLMs, but it charges a harmless typo exactly as much as a silently corrupted name or number. The authors propose semCER, which assigns every grapheme edit a severity score from an LLM judge while preserving CER's alignment and denominator, so that setting every severity to one gives back CER exactly. The goal is a metric that both scores a whole transcription and ranks the individual errors a reader should check first.

Key Contributions

1. An order over errors, not just a score. Sorted by semCER's charge, the edits readers rate most severe come near the top of the list, an order CER cannot produce; the charge tracks readers' ratings beyond what the edit's type predicts and, in English and Chinese, beyond whether normalization would remove the edit.
2. Reader validation where semCER and CER disagree. On pairs drawn where CER and semCER ranked two transcriptions in opposite order, English and Chinese readers side with semCER on 71% and 78% of pairs, against 58% and 60% for normalized CER; Vietnamese readers side with the two equally often. Readers of all three languages side with semCER significantly more often than with a three-class charge.
3. Evidence that an equal edit count hides unequal damage. The judge's charge for an edit varies across 16 languages with the price of the smallest edit, which costs almost nothing in Spanish and nearly a wrong letter's severity in Telugu; where a general VLM and EasyOCR tie on edit count, the VLM's edits cost more in Chinese and Vietnamese and less in English, French, and Spanish.
4. A locally runnable scorer. A 0.8B student model distilled from the judge, trained with no human rating, comes within 0.03 of the judge's agreement with readers and orders nine in ten clear edit pairs as the judge does; it is released with the 61 million judged edits and the aggregated reader ratings.

Main Findings

• Ordering single edits. On the single-edit pool, sorting by severity scores 0.58 to 0.81 on the top-of-list share across English, Vietnamese, and Chinese, and its top 10% catches 81 to 92% of the damage that the readers' own top 10% catches. Sorted by CER's per-edit charge, 1/m, the list scores about zero, the random baseline, with its top 10% catching 29 to 55%, a gap significant in all three languages.
• Severity adds beyond edit type. For the y-to-v swap in strategicallv, the corpus-mean type table charges 0.59, the line-reading judge charges 0.25, and readers give 0.20. Controlling for the type table, the severity still tracks readers in all three languages.
• Severity adds beyond normalization. Pooled over bands, the filter label correlates with readers about as well as the severity does, but within a band the severity still adds agreement once the label alone, or the label and the type table together, are controlled for; its intervals lie above zero in English and Chinese.
• Reading the distribution beats reading the mode. Reading the judge's full probability distribution over the rubric rather than its most likely level adds agreement with readers in Vietnamese and Chinese.
• Whole-transcription scoring goes the other way. Normalized CER's Spearman's rho against readers is 0.93, 0.90, and 0.76, against semCER's 0.89, 0.87, and 0.76, so semCER's value lies in ranking within and between close transcriptions rather than in scoring a document.
• Normalization-proof pairs. Among the 173, 163, and 146 pairs per language with a strict majority, the 22, 30, and 30 normalization-proof pairs are where normalization leaves CER's choice in place: semCER agrees with readers at 0.77 (English) and 0.83 (Chinese) there, against 0.09 and 0.47 for normalized CER; in Vietnamese both agree on about half the pairs. The English and Chinese margins hold under the judge's earlier severities, the distilled scorer, and a second judge.
• Unequal cost by language. The judge charges a wrong letter between 0.63 and 0.81 across scripts, but a wrong mark anywhere from nearly nothing in Spanish to almost a wrong letter's severity in Telugu. Mean severity per edit follows the grapheme load with a rank correlation of 0.75 across the 15 languages with marks, from 0.26 in English to 0.74 in Telugu.
• Unequal cost by system. In German, French, Spanish, and Vietnamese the share of letter errors that form a real word rises from the classical pipelines through the document-specialized VLM to the general VLMs; classical errors stay within 0.10 of the random rate, while general VLM errors form real words above chance in every language and domain but one. The judge charges a general VLM's letter error more than a classical pipeline's in every language where both can be measured, except Spanish.
• The distilled student is close. It correlates with reader ratings at 0.79, 0.63, and 0.80 in English, Vietnamese, and Chinese, against the judge's 0.81, 0.66, and 0.82, and its severity differs from the judge's by a mean absolute 0.04 per edit.
• Worked example. In Figure 2, a general VLM's transcription line makes six edits, giving CER 0.171 over 35 graphemes, while the severities sum to 1.90 and give semCER 0.054.
• Coarser charges trail. The three-class class charge trails semCER significantly in every language on the preference pool, as does the type table, and a single whole-line judgment trails semCER significantly in English and Vietnamese.

Methodology in Plain English

The authors define edits over extended grapheme clusters (Unicode Standard Annex #29 under Normalization Form C), so an accented letter or a base letter plus tone mark counts as one unit in every script and no word segmentation is needed. They run an optimal Levenshtein alignment, break ties toward substitutions and then deletions, and keep CER's denominator, the reference length, so charging every edit one reproduces CER. Each aligned edit is sent to an LLM judge (DeepSeek-V4-Flash Preview at temperature 1) that sees the full reference and output with the edit bracketed in context and returns one of five rubric levels, from 0 for a cosmetic change to 4 for an error that reads perfectly, is false, and leaves no signal. Instead of taking the most likely level, the severity is the expected rating under the judge's renormalized probabilities over the five digits, which lets two edits at the same level differ. When the judge returns no digit or puts less than half its probability mass on the five digits, it is asked once more and otherwise charged one.

The judge was applied to the outputs of seven systems (two classical pipelines, one document-specialized VLM, four general VLMs) over 25 public datasets in 16 languages, totaling 61 million edits and 229,395 transcription lines after filtering. Three baselines use no judge (CER's own per-edit charge, a filter label, a real-word label); a type table uses the judge's corpus but not the line; and two coarser readouts use the same judge (a three-class charge and a whole-line judgment). Human readers of English, Vietnamese, and Chinese rated three pools: about 85 single-edit items per language, about 180 whole transcriptions per language, and about 175 preference pairs per language drawn from disagreements. Comparisons use 95% percentile bootstrap intervals from 10,000 draws, with p-values from bootstrap tails and Holm correction at alpha = 0.05.

Why This Matters

Impact on research. The paper separates two jobs that CER currently performs at once: scoring a whole transcription and telling a user which errors matter. It argues that normalized CER should keep the first job and semCER should take the second, and it supplies the judged corpus, reader ratings, and a local scorer so others can test the claim without paying for API calls.

Real-world applications:

• Checking OCR output for libraries, archives, and publishers, where sorting pages by semCER prioritizes the errors that destroy names, numbers, and entities.
• Document AI and VLM evaluation, where a model that writes fluent wrong words is currently rewarded by a metric that counts edits.
• Digitization of low-resource and non-Latin scripts, where the price of the smallest edit varies sharply by writing system and by the availability of diacritics and tone marks.
• Downstream retrieval and question answering over scanned text, where a low CER can hide decision-critical corruption.

Industry relevance. Judging the corpus cost about $2,000 and the distilled scorer trained for about 30 hours on one A100, but the shipped 0.8B student runs locally, so the metric can be computed without tying scores to a provider's served checkpoint. The authors release severities and ratings under CC BY 4.0 and the scorer's code and weights under Apache-2.0.

Future Directions

• Expand reader validation beyond three languages. Readers validated the severity in English, Vietnamese, and Chinese; in the other thirteen languages the grapheme load comes from the judge alone, and the judge's severities may shift with a language's resource level. The lowest-resource languages in the corpus all use non-Latin scripts, so resource level and script cannot be separated.
• Test other scripts and units. The authors have not tested whether graphemes are the right unit in Thai, which writes no word boundaries, or in the abjads, which leave vowels unwritten.
• Connect semCER to downstream tasks. The paper leaves semCER's value for downstream tasks to future work, and notes that recoverability may not be the right construct, since the rubric's levels depend on whether an output is a valid word.
• Address judge versioning. The provider replaced the judge's checkpoint mid-study, so a severity is comparable only within one checkpoint; the local student avoids this boundary at the cost of approximating the judge.

Target Audience

Researchers and engineers building or evaluating OCR systems and vision–language models, benchmark and metric designers in NLP, and practitioners who need to triage transcription errors at scale rather than only score them. Readers who want a single-number leaderboard metric will find that the paper recommends normalized CER for that role; readers who need to know which errors to check first, or to break ties between systems that produce the same edit count, are the intended users of semCER.

Authors’ abstract

Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.