Towards Robust Mathematical Reasoning

AI summary

Overview

Research area: Natural Language Processing — specifically evaluation benchmarks for large language models' mathematical reasoning, at the level of the International Mathematical Olympiad (IMO).

Technical level: Intermediate. The paper is readable without specialist mathematics, but it assumes familiarity with how LLM benchmarks and autograders are constructed and scored.

Scope: The paper introduces IMO-Bench, a three-part benchmark suite (short-answer, proof-writing, and proof-grading), reports accuracy for a wide set of public and internal models, and validates LLM-based autograders against expert human graders.

What This Paper Is About

Popular math benchmarks such as GSM8K, MATH, and AIME have reached saturation, making them poor at distinguishing model capabilities, and benchmarks that only check final short answers can reward answer-guessing rather than genuine reasoning. The authors build a benchmark suite pegged to IMO-level difficulty — vetted by a panel of IMO medalists (who together won 10 gold and 5 silver IMO medals) and mathematicians — that tests not only correct answers but also proof writing and the grading of proofs. The goal is to establish "north-star" metrics that measure robust mathematical reasoning instead of short-answer matching.

Key Contributions

1. IMO-AnswerBench: 400 diverse Olympiad problems with verifiable short answers, drawn from national, regional, and international contests, evenly split across Algebra, Combinatorics, Geometry, and Number Theory (100 problems each) and across four difficulty tiers (pre-IMO, IMO-Easy, IMO-Medium, IMO-Hard). Problems were robustified — paraphrased, re-named, reformulated, numerically altered, and/or given distractors — to avoid memorization.

2. IMO-ProofBench: A 60-problem proof-writing benchmark split into a basic set (30 problems, pre-IMO to IMO-Medium) and an advanced set (30 problems, up to IMO-Hard), the latter comprising 5 complete IMO sets of which 3 are novel. It ships with detailed grading guidelines and a four-level rubric (Correct = 7, Almost = 6, Partial = 1, Incorrect = 0), with human experts free to assign any integer score from 0 to 7.

3. IMO-GradingBench: 1000 human gradings of solutions to advanced IMO-ProofBench problems, framed as a four-way classification task, designed to advance automatic evaluation of long-form answers.

4. Two validated autograders: AnswerAutoGrader and ProofAutoGrader, both built on Gemini 2.5 Pro, shown to correlate strongly with expert human evaluations.

Main Findings

• IMO-AnswerBench leaderboard: Gemini Deep Think (IMO Gold) reached 80.0% overall accuracy, surpassing the best non-Gemini model (Grok 4, 73.1%) by 6.9% and the best open-weight model (DeepSeek R1, 60.8%) by 19.2%. GPT-5 scored 65.6% and Kimi-K2-Instruct 45.8%. Results were averaged over 8 runs, except Gemini 2.5 Deep Think and Gemini Deep Think (IMO Gold), which were single runs.

• Combinatorics is the hardest category: Models generally performed worst in Combinatorics across IMO-AnswerBench.

• Robustification hurts performance consistently: Every model scored lower on the robustified problems than on the original ones, with gaps as large as 11.2% for o4-mini (high reasoning). Table 9 shows originals versus robustified versions, e.g. Grok 4 at 75.0% versus 73.1% and o3 at 63.9% versus 61.1%.

• Proof writing is much harder than short answers: On basic IMO-ProofBench, Gemini Deep Think (IMO Gold) scored 89.0% while most models scored below 60%. On advanced IMO-ProofBench, all non-Gemini models scored below 25%; Gemini Deep Think (IMO Gold) scored 65.7%, beating the best non-Gemini model (Grok 4 (heavy), 23.3%) by 42.4%. The best open-weight model scored 7.1%.

• Evidence of overfitting to known problem sources: Grok 4 (heavy) scored 76.2% on USAMO 2025 problems but only 11.1% on novel problems; o3 showed 52.4% versus 15.1%, and Gemini 2.5 Pro with (Huang & Yang, 2025) 52.4% versus 17.5%. Gemini Deep Think (IMO Gold) was comparatively balanced at 69.0% on USAMO and 61.1% on novel problems.

• AnswerAutoGrader is near-human: Its predictions against human grades on IMO-AnswerBench gave 99.6% agreement on the negative class and 98.5% on the positive class, reported as overall accuracy of 98.9% on the positive (correct) class.

• ProofAutoGrader correlates strongly with experts: Pearson correlation coefficients of 0.96 (basic) and 0.93 (advanced) across 14 public models, and 0.87 across 170 internal systems from the IMO-gold effort. A confusion matrix over 840 model solutions showed most misclassifications occurred between the Incorrect and Partial classes. Known weaknesses include missing high-level logical errors and being overly punitive toward unconventional but correct solutions.

• Proof grading remains hard for models: On IMO-GradingBench, the highest accuracy was o3 at 54.0%, while Gemini Deep Think (IMO Gold) achieved the best Mean Absolute Error of 18.4%. Because the rubric collapses to scores of 7, 6, 1, and 0, the best possible ("golden") MAE is 3.9% rather than 0%.

• Grok 4 (heavy) instability: Evaluation on IMO-AnswerBench was aborted after only 117 of 400 responses were received despite multiple hour-long attempts; on IMO-ProofBench only 5 of 30 problems responded after 3 attempts, so its basic-set score is reported as NA, and several models had problems treated as incorrect because of query failures.

Methodology in Plain English

The authors assembled Olympiad problems from national, regional, and international contests and then deliberately modified many of them so that a model cannot succeed by recalling the original problem. Modifications include substitution of variables (for example, rewriting an Austria MO 2017 inequality problem in terms of triangle side lengths), fixing a general parameter to a concrete value (as in a USA TST 2005 combinatorics problem), adding irrelevant "distractor" definitions, and reformulating a problem with a new objective (as in a Czech-Slovak Math Olympiad 2017 problem where the task becomes finding all even integers d for which the number of solutions is even). Because many Olympiad problems ask for proofs rather than numbers, the authors reworked some items so each yields a clear, nontrivial short answer.

For grading, rather than restricting problems to those whose answers can be expressed as SymPy objects (the approach used by FrontierMath), they prompt Gemini 2.5 Pro to extract the final answer from a generated solution and judge it semantically against the ground truth — this is AnswerAutoGrader, and it tolerates differences in wording and formatting. For full proofs, they built ProofAutoGrader, which receives the problem statement, the candidate solution, a reference solution, and grading guidelines, and outputs a score on the four-level rubric. All headline proof results are nevertheless based on expert human evaluation; the autograder was validated by comparing its scores to human scores across many models and internal systems. They then collected 1000 such human gradings into IMO-GradingBench to test how well models can grade proofs given only the problem and proposed solution, with no reference solution or rubric.

Why This Matters

Impact on research: The paper argues that benchmark design determines what capabilities get optimized. By releasing a suite that includes proof writing and proof grading alongside short answers, it gives the field a harder, less gameable signal than AIME- or MATH-style evaluation, and its robustification study empirically demonstrates that perturbation measurably lowers model scores — consistent with findings cited from SVAMP, Lila, the functional MATH variant, Putnam-AXIOM, and MATH-Perturb.

Real-world applications:

• Automated grading of open-ended mathematical work, where the paper's AnswerAutoGrader and ProofAutoGrader provide a scalable alternative to scarce expert graders.
• Mathematical education and Olympiad training, where the benchmark and graders can assess student or tutoring-system solutions at competition difficulty.
• Model-verification pipelines that need to detect when a correct final answer hides flawed reasoning.
• Competitive-programming-adjacent tooling for evaluating whether AI systems can produce rigorous, checkable arguments rather than plausible-looking output.

Industry relevance: The work comes from Google DeepMind and directly supported the gold-level performance at IMO 2025 with Gemini Deep Think. It reports clear separation between frontier proprietary systems, open-weight models, and agentic frameworks such as Gemini 2.5 Pro with (Huang & Yang, 2025), giving labs a comparative yardstick. The demonstration that models can be overfit to problem sources (high USAMO scores alongside low novel-problem scores) matters for anyone selecting benchmarks for model releases.

Future Directions

• Improving proof autograders. ProofAutoGrader can miss high-level logical errors and penalize unconventional but correct solutions; closing these gaps would reduce reliance on expensive human experts.
• Reducing evaluation cost. The authors state that definitive assessments still require human verification, which is costly and hard to source, so cheaper yet reliable automatic grading remains open.
• Preventing benchmark decay. Because IMO-Bench is publicly released, its problems and solutions risk being scraped into future training data, so maintaining benchmark integrity against memorization is described as a field-wide challenge.
• Closing the reasoning gap. Even the strongest model scored 65.7% on advanced IMO-ProofBench and 18.4% MAE on IMO-GradingBench, leaving substantial headroom in sophisticated mathematical reasoning and in judging proofs.

Target Audience

AI researchers and engineers working on LLM evaluation and mathematical reasoning; benchmark designers concerned with contamination and robustness; specialists building automated grading or verification systems; and mathematically trained reviewers (contest problem setters, Olympiad coaches) who need to understand what current models can and cannot do on IMO-level material.

Authors’ abstract

Finding the right north-star metrics is highly critical for advancing the mathematical reasoning capabilities of foundation models, especially given that existing evaluations are either too easy or only focus on getting correct short answers. To address these issues, we present IMO-Bench, a suite of advanced reasoning benchmarks, vetted by a panel of top specialists and that specifically targets the level of the International Mathematical Olympiad (IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench first tests models on 400 diverse Olympiad problems with verifiable short answers. IMO-Proof Bench is the next-level evaluation for proof-writing capabilities, which includes both basic and advanced IMO level problems as well as detailed grading guidelines to facilitate automatic grading. These benchmarks played a crucial role in our historic achievement of the gold-level performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-Proof Bench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4% respectively. We also showed that autograders built with Gemini reasoning correlate well with human evaluations and construct IMO-GradingBench, with 1000 human gradings on proofs, to enable further progress in automatic evaluation of long-form answers. We hope that IMO-Bench will help the community towards advancing robust mathematical reasoning and release it at https://imobench.github.io/.