Skip to content
AI.info

The Pulse

LlamaIndex proposes severity-based scoring for document parsing

LlamaIndex says its ParseBench-2 benchmark scores document-parsing errors by their severity. The company describes a test covering 4,317 page-level cases and cautions that agreement with its scoring policy does not establish agreement with every user.

LlamaIndex proposes severity-based scoring for document parsing

Document-parsing scores should penalize a wrong financial value more heavily than a harmless change in table formatting, under a benchmark design described by LlamaIndex. In a November 19 post, the company set out how ParseBench-2 assigns severity to errors and tests whether its scoring rules respond accordingly.

The benchmark covers 4,317 page-level cases across six dimensions: text, formatting, tables, charts, forms and visual grounding. Complete reference tables replace selected-value checks for all 1,025 charts, allowing evaluation across each data series.

Why severity matters

Document parsers turn material such as financial statements, insurance filings and forms into machine-readable output. A benchmark can test whether a parser transcribes text, preserves reading order, associates table values with the right headers and identifies form fields and their values.

One example in the post illustrates a weakness in the earlier ParseBench-1: an output that truncated a financial value scored 99.2, while a factually correct output with different table markup scored 43.6. A separate table comparison metric, TEDS, also ranked the corrupted output higher, according to the company.

Similarity measures can spread the effect of a serious error across a larger body of correct content, while structural comparisons can penalize equivalent representations. LlamaIndex says the significance of markup depends on the application: explicit table semantics may matter to software consuming the markup, while financial extraction may place greater weight on values and their associations.

Testing the scoring rules

ParseBench-2 assigns errors and harmless variations to critical, major, minor or benign categories, with target ranges for how much each should change a score. Its six test suites contain 330 types of controlled changes to otherwise correct outputs.

Researchers then measure departures from the targets and use coding agents to revise comparison rules, error detectors and weights under human review. Humans specify the intended behavior and resolve ambiguities, while regression checks guide changes to the scorer. For fixed inputs and versions, the resulting scorer is deterministic.

For five dimensions, a page receives the lower of its content-recovery score and a severity cap. Under the default policy, the first detected critical error caps an otherwise perfect page at 90. Visual grounding uses separate rules without that cap.

Judges and customization

The company also tested a language-model judge, Haiku 5.5, on 412 pages of SERFF insurance filings, comparing LiteParse plain-text output with r-1-mini. Each page received three judgments with the presentation order varied.

The team examined the 77 pages on which LiteParse received the higher mean preference score. On 11 pages in that selected subset, the winner reversed across judgments even though the recorded prompt, page image, parser outputs and presentation order were unchanged, the company says. It specifies that the test evaluated its own setup, not ParseArena’s deployed judge.

ParseBench-2 gives its six dimensions equal weight by default. The Metric Lab prototype lets users alter dimension weights and severity penalties, reclassify supported error types, or remove their contribution to the cap. It also lets users inspect how rankings respond, export policies and rescore saved evaluations.

The company cautions that agreement with its severity policy in development scenarios does not establish agreement with all users or generalization to unseen documents. An independent check against human judgments beyond those scenarios is needed, it says.

Sources

  1. Original source · Blog post LlamaIndex llamaindex.ai

Companies in this story

Related stories

Explore

More articles