The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

AI summary

Overview

Research area: AI evaluation and benchmarking; mathematical discovery by large language models; construction problems in combinatorics, coding theory and algebra.

Technical level: Intermediate. The benchmark design and scoring are explained in accessible terms, but the underlying problem families (cap sets, Schur colourings, spherical codes, trifference codes, degree–diameter graphs) assume comfort with finite fields, graph theory and extremal combinatorics.

Scope: The paper introduces the Endless Exam, a benchmark of fourteen parameterised families of mathematical construction problems with 69 evaluation instances, uncapped quality scores, and automatic verification, and reports results for nine models in fourteen tool-free configurations plus three tool-assisted configurations.

What This Paper Is About

Existing AI benchmarks saturate — the paper cites a study finding 29 of 60 language-model benchmarks with high or very high saturation — so scores stop distinguishing leading models, and reaching an already-published human result is treated the same as exceeding it. The Endless Exam addresses both problems by posing mathematical construction tasks where any submitted object can be checked automatically and scored continuously against a reference, with no ceiling at 1, and by varying problem parameters (dimension, graph degree, code length) to generate harder instances indefinitely. The goal is a measurement scale that keeps working before and beyond published mathematical frontiers.

Key Contributions

1. An uncapped measure of mathematical progress. Relative quality places diverse construction objectives on a common scale and preserves the magnitude of improvements beyond a reference, rather than clipping at 1.
2. An expandable suite of construction problems. Fourteen families generate new instances by varying parameters such as dimension, graph degree and code length; 69 instances are evaluated in release bench-v1.0.
3. Automatic verification at large scales. Constructions are verified through their mathematical properties rather than by a model judge, using compact descriptions and algebraic certificates — including submitted codes with over a trillion codewords, where checking the finite factors of a Shannon code certifies that count.
4. A model comparison with dual-verifier agreement. Two independent verifier implementations agree on validity and objective value for all model constructions and all 69 reference constructions, and reject all deliberately invalid test cases.

Main Findings

• No published frontier was surpassed. Across fourteen tool-free configurations, the breakthrough rate on the 30 instances with published frontiers was 0%. Tool-free overall scores span 7.55 to 91.90.
• Tool access changes the picture substantially. GPT-6 Astra at high effort with code and web access raises its overall score from 91.90 to 143.16 at 100% validity; GPT-5.6 Luna rises from 34.85 to 119.35 (valid answers 47 to 66); Claude Opus 5.5 rises from 69.81 to 180.06 (validity 57 to 69). Claude Opus 5.5 achieves the highest overall score, with its advantage concentrated in Heilbronn and the length-64 trifference instance, while GPT-6 Astra achieves the highest mean relative quality on published-frontier instances.
• Models match references without beating them. On the 30 published-frontier instances, tool-assisted Astra, Luna and Opus 5.5 match 29, 23 and 23 references respectively, and surpass none.
• Construction baselines are frequently improved on. On the 39 instances with construction baselines, Astra, Luna and Opus 5.5 improve on the baselines for 33, 25 and 33 instances.
• Gap closed remains small. With tools at high effort, mean gap closed across the 64 instances with proven bounds is 6.3% for GPT-6 Astra, 4.7% for GPT-5.6 Luna and 7.5% for Claude Opus 5.5. The largest tool-free mean gap closed is 1.6%, achieved by GPT-6 Astra at high effort.
• Model size and reasoning effort matter, but not monotonically. On published-frontier instances, Qwen3.5's mean relative quality rises from 0.07 to 0.12 to 0.13 at 4B, 9B and 27B. Raising effort from medium to high increases Astra's mean from 0.66 to 0.76 and GPT-5.6 Luna's from 0.16 to 0.25. Claude Fable 5.1 instead falls from 71.90 at medium effort to 43.73 at high effort, where 32 of its 69 responses exhaust the 128k output budget.
• Quality degrades as problem size grows. For maximum-degree-three graphs, all four tested configurations have lower mean relative quality at diameter 10 than at diameter 4; at diameter 10 the means are 0.30 (Astra), 0.17 (Luna), 0.09 (Qwen3.8-27B) and 0.00 (DeepSeek V4.1 Flash). Spherical-code scores fall between dimensions 10 and 16 for all four models.
• Staying near a reference can still mean being far from a bound. For cap sets, 112 points meet both the reference and the proven upper bound at six dimensions; at twelve dimensions both Astra answers combine two such caps to obtain 12,544 points, a relative quality of 0.97 against the reference of 12,928 points, but only 24.8% of the proven upper bound of 50,571.
• Search budget has limited effect on most baselines. Across the eight families classified as search-resistant, increasing search budget from 10 to 600 seconds improves the baseline by less than 10% on every measured instance; Heilbronn benefits more. Trifference is excluded because its baseline procedure does not use the search-time budget.
• Scores distinguish successive historical breakthroughs. Against the historical eight-colour Schur reference of 5,041 elements from Fredricksen & Sweet (2000), constructions of 5,286 (Rowley, 2021) and 5,362 (Bengone et al., 2026) score 1.049 and 1.064; the benchmark uses 5,362 as its eight-colour reference.

Methodology in Plain English

The authors collected fourteen families of mathematical construction problems — for example, building the largest set of points in a finite vector space with no forbidden pattern, the largest sparse graph with a small diameter, or a code with a specified separation property. Each family comes with a rule for checking whether a submitted object is valid, and an objective to maximise or minimise.

Because the optimal answer is unknown for all 69 main-evaluation instances, each instance has a reference: either a published frontier construction (30 instances) or a verified construction produced by the authors' own baseline algorithms (39 instances). A submission's score is its objective value divided by the reference value, or the reference divided by the objective value when the objective is to minimise. This is why scores above 1 are possible and why the scale is described as uncapped; a score of 100 overall means matching the references on average, computed as (100/69) times the sum of the 69 relative qualities.

For problems where the best possible answer is not known but an upper bound is proved, the authors additionally report gap closed, a logarithmic measure of how far a construction has moved from the reference toward the bound; this is averaged over the 64 instances with proven bounds, with the 5 LABS instances using a conjectured target reported separately.

To make large answers tractable, many families accept a compact description instead of an explicit list — products, lattice orbits, digit sets, difference sets, Cayley generators and algebraic certificates — which the verifier either expands or validates algebraically. In the evaluation protocol, tool-free configurations get one response per instance within a 128k output token budget; a tool-assisted track grants code execution and web access with four CPU threads, 16 GiB of memory and a two-hour wall-clock limit. Invalid answers and generation-budget exhaustion score zero.

Why This Matters

The paper's central claim is that once models reach benchmark ceilings, evaluation must keep offering both finer distinctions and harder tasks. The Endless Exam is designed to do both: its scores remain meaningful above 1, and its parameterised families can be scaled to larger dimensions, graph diameters or codeword lengths as models improve — the source of the name.

Impact on research: it provides an uncapped, automatically verifiable target for AI systems that improve mathematical objects, and a common framework in which results from different systems, families and problem sizes can be compared. It also explicitly positions itself against proof- and answer-based benchmarks (FrontierMath, IMProofBench, IMO-AnswerBench, HorizonMath, MathConstruct and others) by scoring construction quality rather than binary correctness.

Real-world applications (the paper does not spell out commercial uses; these follow from the problem families it evaluates):

• Error-correcting and communication codes, via the Shannon code and trifference code families.
• Faster numerical linear algebra, via bilinear matrix multiplication schemes of the kind Strassen (1969) and AlphaTensor (Fawzi et al., 2022) addressed.
• Network topology design, via degree–diameter graphs that keep sparse networks well connected.
• Signal and vector packing, via spherical codes where pairwise angles must stay separated.

Industry relevance: AI labs need benchmarks that still discriminate between frontier models; the paper reports that alongside GPT-6 Astra, GPT-5.6 Luna and Claude Opus 5.5, the smaller and open-weight configurations (DeepSeek V4.1 Flash, Qwen3.8-27B and Qwen3.5 at 4B, 9B and 27B) are scored on the same scale. The search-budget sensitivity analysis also speaks to teams building discovery systems such as FunSearch and AlphaEvolve, which evolve programs against numerical quality feedback.

Future Directions

• Surpassing published frontiers. The current release reports that none of the 30 published-frontier references is surpassed; the scoring machinery is explicitly built so that future improvements over those frontiers are measured in the same way.
• Scaling the instances. Larger parameter settings can supply more demanding tasks under the same mathematical rules, and future benchmark versions can add larger instances while preserving earlier releases and their references for comparability.
• Extending certificate formats. The paper notes that answer formats and verification budgets currently limit which constructions can be evaluated, and that future versions can extend certificate formats alongside instance sizes.
• Quantifying variability and broadening coverage. The authors used one evaluation per instance and configuration, and state that repeated evaluations would quantify run-to-run variability, which the instance-bootstrap confidence intervals do not measure.
• Community contributions. The benchmark is open source, and the authors invite contributions of new constructions, task families and verifier improvements.

Target Audience

Researchers and engineers working on AI evaluation, benchmark design and mathematical reasoning at frontier-model labs; mathematicians interested in computational attacks on open construction problems in combinatorics, coding theory and algebra; and developers of automated discovery systems that optimise mathematical objects against numerical quality feedback. The paper assumes a reader comfortable with extremal combinatorics and finite-field constructions but explains its scoring and protocol in self-contained terms. Code and data are released at https://github.com/ricolalemon/endless-exam.

Authors’ abstract

We introduce the Endless Exam, a benchmark for measuring mathematical progress from today's models toward artificial superintelligence through fourteen parameterised construction families. Each submitted object is checked automatically for validity and assigned a relative quality score against a published frontier or construction baseline, without capping improvements at $1$. The families draw on open mathematical problems for long-term targets and generate new instances at larger parameters, where compact certificates keep large constructions verifiable. Across eight models evaluated on 69 distinct instances, continuous quality scores distinguish performance even though no evaluated system surpasses a published frontier. Size-quality curves show how construction quality changes as problem size increases. We release the generators, verifiers, references, model responses and analysis to support continued measurement before and beyond human frontiers.