Document Vision and Layout Understanding

Each layer can fail independently and should be evaluated separately. The first layer is not a formality. The digit substitutions David Kriesel published in 2013 happened inside the JBIG2 compression of Xerox WorkCentre and ColorQube scanners, before any recognizer saw the page. A document whose text layer was later read with perfect character accuracy still carried an 8 where the paper said 6. Page capture is where evidence is created or destroyed. Every layer above it can only be as faithful as what it was handed.

Architecture choice depends on layout variability, annotation supply, and required evidence. The three options are not equally good at recovering structure, and that has been measured rather than argued. IBM Research Australia scored the field against its own table test set, PubTabNet, in 2020. The best off-the-shelf PDF tool of the coordinates-and-rules kind, Camelot, reached 73.0% TEDS. Tabula reached 67.9, Adobe Acrobat Pro working on images 53.7, PDFPlumber 40.4. The authors' own EDD model, working from the page image alone with no PDF text layer to lean on, reached 88.3%. The handcrafted pipeline is the one you can inspect line by line. It is also the one giving up roughly fifteen TEDS points to a model that was never told where the rules were.

A naïve top-to-bottom pass can merge columns, attach captions to the wrong figure, or read a footer inside a table. That pass has a number. For EMNLP 2021, Wang and four colleagues built ReadingBank: 500,000 document pages, 400,000 training, 50,000 validation, 50,000 test. Its ground-truth reading order was recovered from Word XML metadata rather than guessed by a human. Against that ground truth, sorting the words of a page left-to-right and top-to-bottom scores 0.6972 average page-level BLEU, with an ARD of 8.46. Their LayoutReader model scores 0.9819 BLEU and 1.75 ARD on the same pages.

The result is not confined to their own model's output. Re-ordering the text lines emitted by Tesseract raised BLEU from 0.7532 to 0.9360, and for a commercial OCR API from 0.8530 to 0.9430. In both cases the characters were already recognised. Only the sequence was wrong. The abstract puts it plainly: “It performs almost perfectly in reading order detection and significantly improves both open-source and commercial OCR engines in ordering text lines in their results in our experiments.”

So reading order should be modeled and tested as its own output, with its own score. Use explicit relations such as “follows,” “belongs to row,” and “labels field.” Preserve page coordinates so reviewers can trace extracted values back to evidence.

Tables are relational structures, so cell-level OCR is only the beginning. For years the field's own standard scoring could not see the difference. The IBM team behind PubTabNet tested the adjacency-relation metric used in table competitions by damaging tables on purpose. Shifting 90% of the cells in a table's first row still leaves its F1 near 80%. The tree-edit-distance metric they proposed, TEDS, drops about 60 points on the same wrecked table. Their diagnosis of the older metric is that “as it only checks immediate adjacency relations between non-empty cells, it cannot detect errors caused by empty cells and misalignment of cells beyond immediate neighbors”. PubTabNet itself — 568k table images with HTML ground truth, released in 2020 — is what made the comparison possible, and TEDS bought its EDD model a 9.7% absolute margin over the prior state of the art.

Ground truth suffers from a version of the same disease. Microsoft released PubTables-1M in 2022: 947,642 fully annotated tables, with bounding boxes for every row, column and cell, blank cells included, in both image and PDF coordinates. Its authors named the defect they had to remove before the annotations were usable: “It also addresses a significant source of ground truth inconsistency observed in prior datasets called oversegmentation, using a novel canonicalization procedure.” One logical cell annotated as several is oversegmentation. A model trained against it learns to reproduce it.

A useful test plan reports where information is lost, and each step needs a metric that can actually detect the failure it is meant to catch. Step 2 is where that bites twice. Score table structure with a measure that reads the whole grid: TEDS, or GriTS, which compares the ground-truth and predicted matrices of grid cells rather than counting independent cells. GriTS came from the PubTables-1M authors in a separate 2022 paper, and their dataset paper uses it to report table structure results. Then set the target for region detection against what people achieve rather than against a perfect score. DocLayNet's triple-annotated pages put human-versus-human agreement at 82–83% mAP@0.5–0.95 overall and 60–72% on Title, so a region detector reported at 85% is not obviously worse than a second annotator. Step 2 should also carry a reading-order score of its own. The left-to-right, top-to-bottom baseline sits at 0.6972 BLEU on ReadingBank, which is what a pipeline silently accepts when it never measures order at all.

Use a three-page supplier invoice with one table, handwritten corrections, and a subtotal carried to the next page.