The Pulse
First Proof’s Top System Solves 9 of 10 Research-Math Problems
First Proof’s third benchmark found that its top-performing system solved nine of ten unpublished research-math problems, while the open-weight entry produced no submission free of major errors. The results also highlight the cost of strong
AI.info Team · Written with AI ·
First Proof’s top-performing system solved nine of ten research-math problems in the project’s latest test; its entry built on an open-weight model produced no submission free of major errors. The results, published October 14, 2026, put a measurable gap between the tested systems—but also show that more capable answers came with sharply higher costs.
The third batch tested five AI systems on ten unpublished problems drawn from active mathematical research. Reviewers found that at least one system solved each problem, setting aside shortcomings in citations and exposition. The only open-weight entry, based on Moonshot AI’s Kimi K3, did not deliver a solution without major errors.
Five systems, ten unpublished problems
The problems covered fields including geometry, topology, statistics and dynamics. Each had a human-written solution of no more than 12 pages, and organizers gave each system 24 hours to work without human input. About 30 expert reviewers assessed the anonymized submissions for mathematical correctness, clarity of exposition and attribution of prior work.
The comparison tested systems, not just the underlying models. The entries included direct tests of OpenAI’s GPT-6 Astra and Anthropic’s Siege harness, alongside ProofCouncil and MerLean, which academic teams built around multiple models. A fifth entry, Humanize, used the open-weight Kimi K3 model. First Proof defines a harness as software that can break work into steps, check results and make repeated model calls; its own release notes that the same model can perform differently in different harnesses.
ProofCouncil’s nine solutions cost $1,888
ProofCouncil answered nine problems correctly at a reported cost of $1,888. A direct GPT-6 Astra test answered seven correctly for $36. The release says both improved by two correct answers over the previous batch; those earlier runs cost $3,200 for ProofCouncil and $117 for OpenAI.
The Kimi K3-based system cost about $2,000, yet every submission contained major errors, according to the organizers. Reviewers judged it generally less capable than GPT-5.2 Pro, which First Proof identifies as the leading frontier model in December 2025. The result concerns that particular open-weight model and its test setup; it does not establish how every open-weight model would perform.
Correct answers still lacked clear proofs
Getting the mathematical result right did not guarantee a strong written solution. Only four of GPT-6 Astra’s ten answers received adequate citation assessments, compared with seven for ProofCouncil, which used Astra as a base model. Even ProofCouncil, the highest-scoring system, received negative average exposition scores on a third of its responses.
The gap also raises an access question: a system that improves results by using more computation can be much more expensive than a single model query. The report’s authors—Mohammed Abouzaid, Nikhil Srivastava, Rachel Ward and Lauren Williams—write:
By leveraging more compute, it is possible to significantly improve on the performance of a single query to GPT-6 Astra – but this comes at significant financial cost, leading to concerns that access to computational resources may become a barrier for basic research in mathematics.
First Proof plans separate tests in January
First Proof says its next batch is planned for the first half of January 2027. Organizers intend to test frontier and open-weight systems separately, aiming to identify how open-weight systems fail. The project has published the benchmark problems, solutions, referee reports, system logs and costs, allowing readers to inspect the evidence behind the scores.
The October results leave two concrete questions for that next round: whether a different open-weight model or harness can narrow the gap, and whether the gains from multi-model systems can be made affordable to researchers who lack access to large computing budgets.