Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs

AI summary

Overview

Research area: Document understanding / computer vision, specifically PDF parsing and evaluation of mathematical formula extraction, with an emphasis on benchmarking methodology and LLM-based semantic evaluation.

Technical level: Intermediate — readers should be comfortable with concepts like LaTeX, PDF parsing difficulties, LLM evaluation, and standard similarity metrics (edit distance, BLEU).

Scope: The paper introduces a synthetic-PDF benchmark with exact LaTeX ground truth and a semantic (LLM-as-a-judge) evaluation pipeline to compare 20+ contemporary PDF parsers on mathematical formula extraction.

What This Paper Is About

Most scientific knowledge today is locked inside PDFs, which were designed for visual printing rather than machine-readable semantics. Mathematical formulas are especially difficult because their meaning depends on two-dimensional layout (fractions, superscripts, subscripts, radicals), and no parser reliably converts them into faithful LaTeX. The paper's goal is to give researchers and practitioners a trustworthy, reproducible way to measure how well different document parsers actually extract formulas — not just whether characters match, but whether the mathematical meaning survives.

Key Contributions

1. A synthetic PDF generation framework. The authors generate PDFs from randomized LaTeX templates with precise, automatically known ground truth, avoiding the trade-offs of manually annotated or source-derived benchmarks. Layout, fonts, columns, languages, and formula placement are all controllable.

2. A robust two-stage formula-matching pipeline. Because parsers emit inconsistent delimiters, ordering, and formatting, the authors combine an LLM-based extraction stage (GPT-5-mini) with a deterministic fuzzy Levenshtein-based validation stage, including a retry mechanism for hard cases.

3. First application of LLM-as-a-judge to formula evaluation, validated by humans. A human study with 250 formula pairs and 750 ratings from 30 evaluators shows the LLM judge correlates with human judgment at r=0.78, versus r=0.34 for CDM and roughly r≈0 for text similarity metrics.

4. Open benchmark, datasets, and leaderboard. The paper releases the wikipedia-latex-formulas-319k dataset, 100 synthetic PDFs containing 1,411 inline and 641 display formulas, all parser outputs on Zenodo, and a public leaderboard covering 20+ parsers.

Main Findings

• Large performance spread across parsers. Top systems score above 9.6/10 (Qwen3-VL-235B-A22B at 9.76, Gemini 3 Pro at 9.75, PaddleOCR-VL at 9.65, Mathpix at 9.64), while traditional rule-based tools and un-specialized models fall far lower (GROBID 5.70, PyMuPDF4LLM 6.67, PyPDF 7.69).

• Small specialized models punch far above their weight. PaddleOCR-VL (0.9B parameters) and Mathpix ($0.005/page) rival or beat much larger general-purpose vision-language models, suggesting that document specialization matters more than raw model scale.

• LLM-as-a-judge dramatically outperforms traditional metrics. Text similarity metrics (Levenshtein, BLEU) show essentially zero correlation with human scores. CDM reaches r=0.34. GPT-5 and GPT-5-mini reach r=0.78, making them the most human-aligned automated metric tested.

• CDM has systematic failure modes. It produces false positives (perfect scores despite structural errors like wrong superscript/subscript placement, or matching semantically distinct symbols) and false negatives (cannot process Unicode output from rule-based parsers like pypdf and pymupdf4llm, which emit symbols instead of LaTeX commands).

• Even top parsers make occasional severe mistakes. High-scoring systems are not error-free; catastrophic formula failures become rarer but do not disappear at the top of the leaderboard.

• A surprising result buried in the table. GPT-5-mini — the model used as the evaluator in the pipeline — ranks 20th out of 21 as a parser (6.61), illustrating that being a good judge is not the same as being a good document parser.

Methodology in Plain English

The authors sidestep the messy problem of annotating real PDFs by generating their own. They first extract hundreds of thousands of LaTeX formulas from English Wikipedia, filter out trivial ones (like x^2+1), and deduplicate. Then they randomly assemble these formulas into LaTeX documents with randomized layout settings — different document classes, fonts, margins, and one- or two-column formats — and add filler text in English, German, French, or Spanish via the Faker library. Each document is compiled with pdflatex and iteratively filled until it reaches exactly one dense page. Because the authors wrote the LaTeX, they know the exact ground truth for every formula.

To evaluate parsers, they feed each PDF to a parser and receive back markdown or text. Matching that output back to the ground truth is the hard part: parsers drop delimiters, reorder content in multi-column layouts, merge formulas, or omit them entirely. The authors use GPT-5-mini to extract formulas in sequential order and flag grouped ones, then verify each extraction with deterministic fuzzy matching (whitespace/backslash normalization plus sliding-window Levenshtein distance), with retries on failures. Successfully matched pairs are scored by an LLM judge on a 0–10 scale for correctness, completeness, and semantic equivalence. The judge's reliability was validated against a human study of 250 formula pairs rated by 30 PhD and CS students.

Why This Matters

Impact on research. Reliable formula extraction is a bottleneck for training large language models on scientific literature and for building structured scientific knowledge bases. The paper shows that existing evaluation approaches systematically misjudge parser quality, and offers a semantically grounded alternative. It also quantifies a concrete cost: one prior effort spent over 5,000 A100 GPU hours just fixing PDF parsing errors for a 625K-paper corpus, without even checking formula correctness.

Real-world applications:

• LLM and scientific corpora construction — deciding which parser to use for datasets like S2ORC-style pipelines, where formula fidelity directly affects downstream model performance.
• Retrieval-augmented generation (RAG) and semantic search over academic literature — an incorrectly parsed fraction or subscript silently corrupts retrieval and reasoning.
• Assistive technology and accessibility — only 3.2% of scholarly PDFs meet accessibility standards; better parsers enable screen readers and other tools to render mathematics meaningfully.
• Practical parser selection — the leaderboard gives engineering teams concrete guidance on which tools to deploy, including cost/performance trade-offs (e.g., Mathpix at $0.005/page vs. free but weak rule-based tools).

Industry relevance. Document AI is a crowded commercial space (Mathpix, LlamaParse, Mistral OCR, Gemini, GPT models). This benchmark provides an independent, reproducible way to compare these paid and open services on a task where marketing claims rarely match measured performance, and it shows that small, cheap specialized models can beat expensive general-purpose APIs.

Future Directions

• Extending to non-formula content. The authors propose applying the same synthetic-generation plus LLM-judge methodology to tables, figures, and complex layouts, which would broaden the benchmark's usefulness.

• Closing the synthetic-to-real gap. The current benchmark uses machine-generated PDFs and excludes scanned documents, historical papers, and publisher-specific artifacts. Validating or extending the framework to these cases is an open problem.

• Improving the judge itself. LLM-as-a-judge is not infallible — it occasionally misjudges identical renderings or misses subtle semantic errors — and it depends on proprietary models. Developing open, cheaper, or more reliable semantic evaluators is a natural next step.

• Understanding why specialized models win. The result that a 0.9B-parameter specialized OCR model beats much larger general VLMs invites further study into what architectural or training choices drive formula-parsing strength.

Target Audience

This paper is most valuable to: (1) ML engineers and data scientists building scientific corpora or training LLMs on academic literature who need to choose a PDF parser; (2) document AI researchers working on OCR, layout analysis, or evaluation methodology; (3) practitioners evaluating commercial parsing APIs who want an independent, cost-aware comparison; and (4) benchmark designers interested in synthetic data generation and LLM-as-a-judge evaluation as general techniques. Readers without background in LaTeX or document parsing will still follow the core argument, but will get more from the paper with some familiarity with those concepts.

Authors’ abstract

Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either exclude formulas entirely or lack semantically-aware evaluation metrics. We introduce a benchmarking framework centered on synthetically generated PDFs with precise LaTeX ground truth, enabling systematic control over layout, formulas, and content characteristics. For evaluation, we apply LLM-as-a-judge to assess semantic equivalence of parsed formulas, capturing mathematical meaning beyond surface-level notation differences. We validate this approach through a human study (250 formula pairs, 750 ratings from 30 evaluators), showing a Pearson correlation of r=0.78 with human judgment, compared to r=0.34 for character-level matching (CDM) and r~0 for text similarity. Our robust two-stage matching pipeline combining LLM-based extraction with fuzzy validation reliably aligns parsed formulas with ground truth despite format inconsistencies across parsers. Evaluating 20+ contemporary PDF parsers across 100 synthetic documents with 2,000+ formulas reveals significant performance disparities, providing actionable guidance for practitioners selecting parsers for downstream applications. Code and benchmark data: https://github.com/phorn1/pdf-parse-bench and https://github.com/phorn1/formula-metric-study