HistGramBench: Measuring Structural Recovery When Multimodal Models Redraw Historical Mathematical Diagrams as TikZ

AI summary

Overview

Research area: Computer vision and multimodal document analysis, specifically the evaluation of vision-language models on graphics program synthesis (image-to-TikZ) for historical mathematical diagrams.

Technical level: Intermediate. The paper assumes familiarity with vision-language models and LaTeX/TikZ, but its central idea—checking whether a redrawn figure preserves the geometric relations a reader would take from it—is explained in plain terms.

Scope (one sentence): HistGramBench introduces a 2,507-figure benchmark of historical mathematical diagrams (1551–1933, eight languages) and a scoring protocol that verifies, on the compiled geometry of a model's own TikZ output, how much of each figure's structural content the model recovers.

What This Paper Is About

Digital libraries hold scanned mathematical books whose lettered diagrams cannot be searched, because a figure's content—which point lies on which circle, which lines are parallel—exists only as pixels. Redrawing such a figure as a short TikZ program would make those relations explicit and machine-readable, but a printed plate has no source code against which a redrawing can be checked. The paper builds a benchmark and a scoring method that judge a redrawing by the typed facts a reader extracts from the figure, evaluated on the model's own compiled drawing rather than against reference code or pixels.

Key Contributions

1. A historical diagram corpus. 2,507 scored figures from 95 public-domain mathematical works printed between 1551 and 1933, in eight languages, drawn from Wikimedia Commons, the Internet Archive, the Göttingen digitization center (GDZ), Numdam, Project Gutenberg and two archival scans. Figures were extracted with the PaddleOCR-VL layout model, passed through image gates and a model screener, deduplicated at a 64-bit perceptual-hash Hamming distance of 6, and split by source group so that no test figure shares a work or near-identical plate with the 150-figure development set (five works). The test side contains 2,507 figures in 87 source groups, the largest holding 669 figures.

2. A dual-writer answer key. 114,403 typed facts (mean 45.6 per figure) across entity, relation and label classes, where relation facts are split into co-exact, depicted exact, marked exact, and three-dimensional/other tiers. A fact enters the key only when two writer models from different providers—GPT-6.1 Sol (OpenAI) and Kimi K2.5 (Moonshot)—both support it under a unanimity rule.

3. A reference-free scoring protocol. Each answer is compiled once with LuaLaTeX (16 outcome codes, only "OK" yields a scored drawing), read as vector geometry, aligned to the key's objects, and checked with predicate tests whose tolerances scale to the drawing's bounding-box diagonal D (ε = 0.01; 1.5° for parallel and perpendicular; 0.015·r_c for tangency). Facts that are judge-only, have an unbound argument, or belong to a drawing with more than 60 primitives go to a three-model judge panel (Qwen3 VL 235B, Gemma 4 31B, Mistral Large 3).

4. A low-cost model evaluation with statistical controls. Six affordable vision-language models—Gemma 4 26B, Llama 4 Maverick, Ministral 3 14B, Gemma 4 E2B, Claude Haiku 4.5 and Nemotron Nano 12B v2—scored with and without page text, analysed at source-group level with Benjamini–Hochberg false-discovery control at q = 0.10, a pre-registered equivalence margin of δ* = 0.140, and ten scoring and weighting variants.

Main Findings

• Modest recovery on the image alone. Under Track A (image only), the six models recover between 0.20 and 0.59 of the answer key; the best recovers about three fifths of the key facts.

• Missing drawings dominate the loss. Between 27 and 49 percent of each model's answers produce no drawing at all, and these non-drawing answers account for 58 to 85 percent of that model's loss. The failure is less about drawing the wrong figure than about not producing a compilable figure.

• Relations are the hardest tier. Relations are the hardest facts for every model, and depicted exact relations (parallel, perpendicular, tangent, collinear, concyclic) are harder than co-exact ones such as incidence and betweenness. Parallels and tangencies are the hardest relations to reproduce.

• Page text does not change recovery. Pooled over the six models, adding up to 1,200 characters of page text (caption first) leaves recovery unchanged within the pre-registered margin: the 95% interval falls inside [−0.140, 0.140]. The 969 plates without page text serve as a control, since on them only the prompt wrapper differs.

• Stable model ranking. The point order of the six models is identical across ten scoring and weighting variants, up to one pair the headline already ties. Every variant that tests separation separates the same three tiers: {Gemma 4 26B, Llama 4 Maverick}, then {Ministral 3 14B, Gemma 4 E2B, Claude Haiku 4.5}, then {Nemotron Nano 12B v2}.

• Robustness of the headline. Removing the largest source group (669 figures) moves no score by more than 0.002. Judge self-agreement when re-answering its own development calls was at least 0.867, against a required 0.8.

• The key is accurate but deliberately incomplete. The unanimity rule keeps 84 percent of entity facts, 79 percent of label facts and 55 percent of relation facts in the combined writer lists—about 40 percent for tangent_to, collinear and intersect_at. Keys retain roughly two-thirds of combined facts (45.6/67.5). Contestants recover relations that both writers generated more often than relations only one generated, at 0.62–0.88 versus 0.45–0.73 across models.

• Key validation evidence. On the 150 development figures, a combined fact was supported on average by 0.835 of the two writer families. In an earlier development run using Claude Sonnet 5 and Kimi K2.5, 170 of the 172 key facts mappable to ground truth were true; 76 percent of all key facts could be mapped, and the key held 51 percent of the true facts.

• Composition skew matters and is reported. Under equal weighting of source groups, the 56 single-work journal groups (mostly the Nouvelles annales de mathématiques, held by Numdam) carry 64 percent of the headline weight while holding 142 figures, and the 30 single-figure groups account for 34.5 percent. The paper therefore reports the headline without the Numdam groups and with each figure weighted equally.

• Taxonomy of the corpus. 2,079 geometry figures (84.4% of headline weight), 404 graphs and plots (15.0%), 24 other (0.5%); 2,365 book figures versus 142 journal-article figures; 1,538 figures with page text and 969 without; 824 crops touched by ink at the border.

Methodology in Plain English

Building the corpus. The authors started from page scans held by public repositories. A layout model (PaddleOCR-VL) proposed figure regions; automated image gates then removed crops that were too small, photographic or mostly text, and split multi-panel regions. A model screener admitted a crop only if it showed a legible mathematical figure that TikZ could express, rejecting when in doubt. Near-identical crops were clustered and represented once, and works were linked into source groups that never straddle the development/test split—because figures from one book share a printer, an engraver and drawing conventions, and treating them as independent would overstate precision.

Building the answer key. Two writer models from different providers independently listed the objects in a figure and the facts relating them. Their object lists were matched by label, position and shared relations (differently labelled objects were never merged), and each writer then voted true, false or abstain on every fact in the combined list. A fact entered the key only if both supported it and it passed consistency, ambiguity and binding checks. Four gates per figure required at least three entity facts and three relation facts, contradictions in no more than 10 percent of combined facts, and unidentifiable objects in no more than 20 percent of matched objects; 2,507 of 3,000 carded test figures passed.

Scoring a redrawing. A model returns one fenced LaTeX block with one tikzpicture. The harness compiles it once. If it compiles, the harness reads the resulting page as vector geometry—segments, circles, arcs fitted to Bézier runs, filled regions, placed text—and aligns those objects to the key's objects using label proximity, structural propagation, uniqueness, then subgraph matching or optimal assignment. Each key fact is then tested on the drawing: is the drawn point labelled C within ε·D of the drawn circle? Are two lines within 1.5° of parallel and not overlapping? Because every distance is measured relative to the drawing's own diagonal D and angles are scale-free, the test is indifferent to translation, uniform scale, line width, font and colour—exactly the invariances the prompt promises.

Handling judgement calls. Some facts cannot be settled geometrically: judge-only predicates (regions, curves, solids, 3D relations), facts whose objects did not bind, and drawings with more than 60 primitives. These go to a panel of three judges from families disjoint from the writers' and from the contestant's own family. Judges decide about a third of verdicts on valid drawings. Three contestants (Gemma 4 26B, Gemma 4 E2B, Ministral 3 14B) have only two eligible judge families, so the first eligible family alone decides their judged facts; those two families disagree on 18 to 27 percent of judged facts, and the paper bounds the effect by counting every such fact as absent and then as present.

Aggregating. A figure's score is the share of checkable key facts found present, multiplied by 1 if the answer compiled and 0 otherwise. Facts whose objects are deterministically absent from the drawing are set aside rather than charged again—this removes 21 to 38 percent of relation facts on valid drawings—so a single missing object costs one entity fact, not a chain of derived ones. Scores are averaged within each source group and then across the 87 groups, with confidence intervals treating group means as observations. Model comparisons use paired t tests on group-mean differences, corrected across 15 pairs with Benjamini–Hochberg at q = 0.10.

Why This Matters

Impact on research. The paper isolates a genuinely new evaluation condition: a historical source image, no reference code, and deterministic relation and label checks against the compiled geometry of the model's own output. Prior image-to-code benchmarks score appearance on modern material with known code; historical-diagram work recovers primitives and text regions but does not score reconstructions. HistGramBench also imports the exact/co-exact distinction from the philosophy of mathematical practice (Manders; Netz; Hamami et al.) into a machine-scored benchmark, which gives diagram-reasoning evaluation a principled vocabulary rather than an ad hoc one. Its finding that most loss comes from answers that never produce a drawing reframes the problem as one of output validity, not only geometric fidelity.

Real-world applications:

• Digital mathematics libraries (Numdam, GDZ, Internet Archive, Project Gutenberg, Wikimedia Commons) that currently store diagrams as unsearchable page images and want queryable, editable figure objects.
• History of mathematics and science research that needs to compare diagram conventions across editions, printers and centuries—including cases like Byrne's 1847 colour-coded Euclid, whose 73 figures fall largely outside the label vocabulary.
• Humanities computing and scholarly editing, where redrawing a manuscript or printed figure is an interpretive act that can erase information (Saito and Sidoli), and a scored redrawing makes that interpretation inspectable.
• Mathematics education and textbook digitization, where a figure's incidence, betweenness and containment relations are exactly what a student or a tutoring system needs to reason over.

Industry relevance. The benchmark speaks directly to document-AI and scientific-publishing pipelines that already handle text and formulas but leave figures behind: the 27–49 percent non-compiling rate is a concrete reliability signal for anyone deploying image-to-TikZ generation in production. It also offers evaluation tooling—reference-free, invariance-aware, judge-bounded—for teams building vectorization, chart-to-code or slide-to-code systems, and it gives model developers a diagnostic split between "cannot draw" and "draws the wrong structure."

Future Directions

• Closing the validity gap. Since 27 to 49 percent of answers produce no drawing and these account for 58 to 85 percent of the loss, the largest single gain available is making models emit reliably compilable TikZ rather than improving geometric fidelity on drawings that already compile.

• Exact relations and non-synthetic figures. Parallels and tangencies remain the hardest relations, and the 404 graphs and plots are scored only on the parts of the vocabulary that apply to them. Extending the fact vocabulary to axes, curves and asymptotes at the same level of rigour, and to conventions such as Byrne's colour coding, is open.

• Reducing reliance on judges and completing the key. Judges decide about a third of verdicts on valid drawings, and the unanimity rule makes the key incomplete by design—the paper scores recall of the key, and a correctly drawn fact absent from the key may still be true of the figure. Human annotation or a higher-recall key construction would let precision be scored as well.

• Human validation and construct validity. The key's accuracy has been checked on synthetic diagrams and development figures, but not reported against a human-annotated sample of the 2,507 test figures; nor does the paper report whether historians agree that the three fact tiers capture what a reader takes from a printed plate.

Target Audience

Researchers evaluating multimodal and vision-language models on structured outputs; benchmark designers working on graphics program synthesis, image-to-code and vectorization; digital library and digital humanities teams pursuing machine-actionable historical collections; historians and philosophers of mathematics interested in diagrammatic reasoning and in what a redrawing preserves; and engineers building document-AI pipelines for scientific and historical print. Readers need some familiarity with LaTeX/TikZ and with standard benchmark reporting, but the framing of the problem is accessible to non-specialists.

Authors’ abstract

Digital libraries preserve mathematical books printed from the sixteenth to the twentieth century, but the lettered diagrams in them, the figures on which Euclidean proofs depend, remain scanned images whose content cannot be searched. Redrawing such a diagram as a short drawing program would record which points lie on which lines and circles. But a printed plate offers no source code against which the redrawing could be checked. We introduce HistGramBench, a benchmark of 2,507 diagrams from 95 works printed between 1551 and 1933 in eight languages. For each diagram, two AI models from different providers agree on the facts a reader takes from the figure, such as which lines are parallel or where two lines meet, and every redrawing is checked against those facts on the model's own drawing. We evaluate six affordable AI models. The best recovers about three fifths of these facts, and most of what the models miss comes from programs that never produce a drawing. Parallels and tangencies are the hardest relations to reproduce. HistGramBench gives historians and librarians a way to measure when such tools can make old mathematical diagrams searchable.