OCR and Scene Text Recognition

End-to-end products often combine detection, rectification, recognition, and validation.

One whole stage of that pipeline disappeared in 2006. Connectionist temporal classification trains a recurrent network “to label unsegmented sequences directly”, as Graves and colleagues put it. Before it, training data had to be pre-segmented by hand and the network outputs post-processed into label sequences. Tesseract followed. Its version 4 “adds a new neural net (LSTM) based OCR engine which is focused on line recognition”, with official models for more than a hundred languages.

Every stage is also a place where the page can change, and not only the one that does the reading. The rest of this lesson works through failures that were documented, dated and in two cases regulated. A compression step that rewrote digits before any recognizer saw them. A field-extraction step that failed on receipts whose characters had been read correctly. A confidence threshold that sent 841,260,847 mail images to human typists in one year. A storage step that kept a copy of everything scanned on a disk inside the machine.

Text recognition difficulty depends on how writing is produced and captured.

Handwriting is the case where the source of variation is a person, and the reference corpora are sized accordingly. NIST Special Database 19, re-released in a second edition in September 2016, holds “Handprinted Sample Forms from 3600 writers, 810,000 character images isolated from their forms, ground truth classifications for those images, reference forms for further data collection, and software utilities for image management and handling.” Cohen and colleagues counted the same collection independently when they derived EMNIST from it: “Both the By Class and By Merge hierarchies contain 814,255 handwritten characters consisting of a suggested 731,668 training samples and 82,587 testing samples.”

Notice the order of the two numbers. 3,600 writers come before 814,255 hand-checked characters. A printed-document corpus is counted in pages, because the fonts repeat across all of them. A handwriting corpus is counted in writers first. The writer is the thing that varies, and a thousand pages from one hand would not buy the coverage that a hundred hands do.

A language model can prefer common words over visually plausible rare strings. That helps prose. But it may damage serial numbers, names, codes, and mixed alphanumeric fields.

Use field-specific constraints and retain visual alternatives. Exact identifiers should not be silently normalized into familiar vocabulary. The same asymmetry runs through the rest of this lesson: an output that reads as clean, fluent text carries no visible signal that a value inside it has been changed.

Character-level averages can hide severe field failures. The recognizer is also not the only stage that can change a character.

In 2013, Xerox WorkCentre and ColorQube machines were silently rewriting digits in scanned documents. In a scanned cost table, “The 65 became an 85 (second column, third line)”, and elsewhere “a 60 in the upper right region of the image became a 80”. No OCR engine had misread anything. The cause sat in compression. JBIG2 “Symbol Coding” stores one image patch and reuses it for every region the pattern matcher judges identical, so one glyph's stored bitmap can be pasted wherever the matcher decided the shapes matched. David Kriesel found the problem on 24 July 2013 and published it on 2 August. Xerox confirmed on 12 August that it had implemented the software bug eight years earlier, and shipped the first patches on 22 August. Its own support article KB0230451 is titled “Random Character Switching When Scanning Using JBIG2 Compression”.

The rule that followed was not aimed at Xerox. Germany's Bundesamt für Sicherheit in der Informationstechnik writes the guideline for scanning that replaces a legal original, TR-03138 „Ersetzendes Scannen“ (RESISCAN). Its requirement A.SC.12 reads: „Verfahren, die zur Bildkompression das sog. „Symbol Coding“ verwenden, DÜRFEN NICHT eingesetzt werden.“ Compression methods that use Symbol Coding must not be used at all. A footnote gives the reason: “Bei ungenauem oder fehlerhaft implementiertem „Symbol Coding“ besteht die Gefahr, dass sich das Scanergebnis semantisch (z. B. durch Vertauschung von Zeichen) vom Original unterscheidet.” The failure mode named there is not noise. It is a scan that differs from the original in meaning, by swapped characters, while looking entirely clean.

Use several levels of error because each answers a different question. The ICDAR 2019 SROIE competition asked all three questions about the same photographs, and the answers came apart.

The organisers built one corpus: “a new dataset with 1000 whole scanned receipt images and annotations is created for the competition” — 600 in the trainval set, 400 in the test set. They graded it at three levels, “Scanned Receipt Text Localisation (Task 1), Scanned Receipt OCR (Task 2) and Key Information Extraction from Scanned Receipts (Task 3)”. The competition ran from 10 February 2019 to 5 May 2019 and drew 29, 24 and 18 valid submissions to the three tasks. 16 of the 29 exceeded 90% Hmean at finding the text. 7 of the 24 exceeded 90% at reading it. Then the third level: “For the Task 3 of key information extraction, we can see only one method achieves Hmean of more than 90% (which is 90.49%), and more than half of the submitted methods achieve Hmean of less than 80%.” Identical images, identical pixels, and the level the buyer actually cares about is the level that collapses. Receipt applications require roughly 99% accuracy, and measured against that the organisers concluded that “even the best OCR method in Task 2 can't deliver the required performance”. Two years later, Microsoft Research's LayoutLMv2 reported the field-extraction score on the same benchmark moving “SROIE (0.9524 to 0.9781)”.

Scene text is graded apart for the same reason. Yao and colleagues note that incidental text, unlike focused text, is not “captured with user’s intention and intervention”.

The last level, simulated review, has a price, and the Australian Newspapers Digitisation Program has published it. A 2009 study measured what a mass digitisation programme gets from the scanner before anyone intervenes: “In a sample of 45 pages to be representative of the libraries digitised newspaper collection 1803-1954, we found that raw OCR accuracy varied from 71% to 98.02% (character confidence, no correction).” Rose Holley banded the outcomes as good at 98-99%, average at 90-98%, poor below 90%. The National Library of Australia's answer to the bottom of that range was to publish the text anyway and let readers repair it line by line. Within twelve weeks of the July 2008 soft launch, “700,000 lines of text have been corrected within 50,000 articles”. The library now reports “524 million lines of text corrected” all-time, alongside 29.1 million newspaper pages online. A 71% page does not produce a slightly worse search index. It produces a correction backlog measured in hundreds of millions of lines.

Scans can contain names, signatures, account numbers, medical data, addresses, and hidden metadata; logging full images or recognized text may create a larger risk than the model itself. The logging is often already happening in hardware nobody classified as a data store.

“Nearly every digital copier built since 2002 contains a hard drive - like the one on your personal computer - storing an image of every document copied, scanned, or emailed by the machine.” Armen Keteyian reported that for CBS News Investigates on 19 April 2010. CBS News bought four used copiers from a New Jersey warehouse holding 6,000 machines, pulled the hard drives out in 30 minutes, and recovered the contents by “downloading tens of thousands of documents in less than 12 hours”. Among them: 300 medical records with drug prescriptions, blood test results and a cancer diagnosis, police sex-crimes complaints, and a narcotics unit's list of raid targets. Nothing was hacked. The disks were bought.

Regulators say the same thing to businesses, without hedging. The U.S. Federal Trade Commission's July 2017 guide to digital copier data security states: “The hard drive in a digital copier stores data about the documents it copies, prints, scans, faxes or emails. If you don't take steps to protect that data, it can be stolen from the hard drive, either by remote access or by extracting the data once the drive has been removed.” Its remedies are encryption and secure overwriting, including before equipment is returned or disposed of.

Collect less, redact when appropriate, restrict access, and say how long anything is kept; review outputs for inadvertent extraction of fields outside the stated purpose. Then ask the retention question again about the scanner, the copier, and the end of the lease.

A representative test set should combine ordinary pages with adversarially ordinary failures. Each item below corresponds to a failure documented earlier in this lesson rather than an imagined one.