Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?

AI summary

Overview

• Research area: Natural Language Processing / automated mathematical reasoning data synthesis using LLM-based code agents.
• Technical level: Advanced (assumes familiarity with LLM agents, test-time scaling, benchmarking, and LLM-as-a-judge evaluation).
• Scope: The paper introduces Code2Math, a multi-agent, code-execution-driven pipeline that autonomously evolves 100 seed math problems into harder, verified-solvable variants and measures the resulting difficulty increase across solvers.

What This Paper Is About

As LLMs approach IMO-level and research-level math, progress is limited by the scarcity of challenging, high-quality problems, which are expensive to create by hand. The paper asks whether code agents, which can run experiments, symbolic computation, and counterexample searches in executable environments, can autonomously evolve existing math problems into new problems that are both solvable and genuinely harder. It builds and evaluates a three-agent system (evolution, solvability verification, difficulty verification) that iterates with multiple rollouts until both verification criteria are met.

Key Contributions

1. A multi-agent framework that decomposes mathematical problem adaptation into three stages: an Evolution Agent, a Solvability Verification Agent, and a Difficulty Verification Agent, with code execution supporting symbolic reasoning and structured exploration.
2. Experiments using five evolution backbones (DeepSeek-Chat, DeepSeek-Reasoner, Gemini-3-Pro-Preview-Thinking, Kimi-K2-Thinking, Seed-2.0-Pro) and six solver models, showing that the framework maintains high solvability while substantially increasing problem difficulty.
3. Three stated findings: code-driven exploration helps discover hidden insights; models can generate challenges beyond their own solving baselines; and stronger difficulty enhancement requires nontrivial computational overhead.
4. A human audit (including an olympiad-level subset reviewed by a mathematics postdoctoral researcher) validating the reliability of GPT-5.2-High as the automatic judge for validity, solver-answer correctness, and relative difficulty.

Main Findings

• High solvability agreement: The external judge certifies between 74 of 90 and 98 of 98 internally accepted problems across evolution backbones. DeepSeek-Reasoner reaches a 94/98 agreement rate, approximately 96%; Gemini-3-Pro-Preview-Thinking reaches 98 of 98.
• Systematic difficulty escalation: Most solver–evolver pairs show lower Evolution-SR than Origin-SR. GPT-5.2-High drops from 70% to 64% on DeepSeek-Reasoner evolutions and to 61% on Gemini-3-Pro-Preview-Thinking and Seed-2.0-Pro evolutions, while Gemini-3-Flash-Thinking drops by up to 32 points.
• Evolver reasoning strength matters: DeepSeek-Chat leaves GPT-5.2-High at 70%, whereas DeepSeek-Reasoner reduces it to 64% and produces larger drops for other solvers such as Gemini-3-Flash-Thinking, indicating that reasoning-enhanced evolution agents introduce structural modifications that transfer across solvers.
• Greater discrimination between solvers: Qwen3-235B-A22B-Thinking has a low 20% seed solve rate and often changes only slightly, whereas Gemini-3-Flash-Thinking starts from a 56% baseline and can lose more than 20 points on multiple evolved sets, exposing robustness gaps less visible on the seed problems.
• Higher reasoning effort: Evolved problems shift the Average Token Consumption distribution to the right, with more high-token and timeout cases, suggesting added difficulty comes from deeper search and self-correction rather than longer wording or heavier arithmetic.
• Computational cost of evolution: Qualified evolution requires multiple rollouts, with average failure counts from 1.56 (Gemini-3-Pro-Preview-Thinking) to 6.55 (Kimi-K2-Thinking). Failures overwhelmingly come from solvability verification rather than difficulty verification, making logical consistency the dominant bottleneck.
• Human audit confirms judge reliability: Of 127 deduplicated evolved problems judged valid by GPT-5.2-High, humans judge 126 as valid (99.2% validity precision). Human–GPT agreement on solver-answer correctness is 1134/1226 (92.5%); exact agreement on the 1–5 difficulty scale is 115/145 (79.3%) and within-one agreement is 139/145 (95.9%), with a mean bias of −0.041.
• Olympiad-level audit: On 35 evolved problems seeded from IMO shortlist, IMO official, and CMO official problems, both humans and GPT-5.2-High judge 34 of 35 as improved under the score ≥3 threshold; the average human difficulty score is 4.03 versus 3.80 from GPT.
• Failure-after-evolution: Among human-valid evolved problems, solvers that solved the seed failed on the evolved version in 166 of 338 applicable solver–problem pairs (49.1%). By evolver, Gemini-3-Pro produces the highest disruption rate at 61.8% and the highest average human difficulty score, 4.55.
• Code ablation: Code-enabled evolution lowers solve rates in seven of nine evolver–solver pairs; averaged across all pairs, the solve rate falls from 25.7% without code to 23.1% with code. The effect is not deterministic, as DS-NT evaluated by DS-NT and Seed evaluated by DS-T show higher solve rates with code.

Methodology in Plain English

The researchers started with 100 seed problems drawn from standard problem books, recent regional exams and competitions, the IMO, and benchmarks including AIME-2024 and AIME-2025, spanning algebra, combinatorics, calculus, sequences, and graph theory. They also selected 6 additional problem pairs (expert demonstrations and pairs created through reverse creation by LLMs) as adaptation and evaluation examples.

Each seed problem is handled by an Evolution Agent that first analyzes the original solution to find the solver's cognitive bottleneck, then freely explores modifications. Difficulty is defined as the "Burden of Discovery" — the challenge of uncovering the key insight — and the agent is instructed to anticipate how an experienced competition solver would attack the problem and then conceal the insights that would create the "Aha moment." A Solvability Verification Agent performs two-stage checking: surface-level error detection, then scrutiny of the proposed solution steps, on the reasoning that a flawless logical chain implies at least one solution path exists. A Difficulty Verification Agent scores the adaptation on a 5-point scale separating "Artificial Complexity" from "Cognitive Depth": scores 1–2 fail (unchanged/regressed path, or difficulty from computational tedium), scores 3 and above pass, with 4–5 reserved for anti-templating adaptations and 5 additionally rewarding mathematical beauty such as deep symmetries.

The whole system follows test-time scaling: multiple rollouts of the Evolution Agent until both verification agents are satisfied, with a maximum of 20 rollouts and a maximum trajectory length of 30 steps per seed. Agents run Python in a Smolagents-based sandbox equipped with libraries such as SymPy, Z3, NetworkX, and itertools, and can run numerical simulations, print sequences, or search for counterexamples. Evaluation uses GPT-5.2-High as an external judge for both solvability and answer correctness; a problem counts as solvable only when the judge and the agent agree. Solvers get up to three attempts per problem and are stopped at a maximum token limit or a 30-minute wall-clock timeout, with all models queried at temperature 0. Reported metrics are Evolution Success Count (ESC), Certified Solvability Count (CSC), Agreement Rate (AR), Solve Rate (SR) on seed and evolved problems, and Average Token Consumption (ATC).

Why This Matters

The work shifts the bottleneck question in math reasoning from "can models solve hard problems" to "can models manufacture hard problems," and provides empirical evidence that code-driven agents can be a viable mechanism for synthesizing high-difficulty mathematical reasoning problems inside scalable computational environments. It matters for research on training-data synthesis, benchmark construction, and self-evolution of reasoning models, and it documents a concrete trade-off between strict dual verification (reliability) and rollout cost (efficiency).

Real-world applications:

• Automated generation of fresh, harder evaluation sets for math reasoning benchmarks, reducing reliance on manual curation and mitigating benchmark contamination.
• Curriculum and training-data production for math tutoring or model fine-tuning pipelines, where validated, solvable problems of escalating difficulty are needed.
• Adversarial testing and robustness auditing of deployed reasoning models, since the evolved problems discriminate between solvers more sharply than the seeds.
• Agentic tooling for researchers and competition organizers who want to probe conjectures, extremal constructions, and combinatorial structures in executable environments.

Industry relevance: organizations building or evaluating reasoning models could adopt this pipeline as an internal difficulty-scaling harness, though the reported failure counts (1.56 to 6.55 per success, with difficult cases often exceeding 10 iterations) indicate meaningful compute costs.

Future Directions

• Improve rollout efficiency and strengthen solvability guarantees, since most failures come from solvability verification rather than difficulty verification.
• Develop more systematic mechanisms for structural synthesis, as the paper notes that code execution currently enables local validation and structural probing but not full structural synthesis.
• Test whether the evolved problems actually improve model performance when used as training data; the paper explicitly states this downstream training value was not verified.
• Scale to a larger and more diverse seed set and conduct more comprehensive human evaluation, given that the current study uses 100 seed problems and a sampled rather than exhaustive human audit.
• Evaluate whether similar exploratory strategies generalize beyond mathematical reasoning.

Target Audience

Researchers and engineers working on LLM mathematical reasoning, synthetic data generation, benchmark design, and agentic systems; practitioners who need to build harder evaluation or training sets for reasoning models; and mathematically literate readers interested in how executable exploration can support problem construction. The paper is most useful to those already comfortable with multi-agent pipelines, test-time scaling, and LLM-as-a-judge methodology.

Authors’ abstract

As large language models (LLMs) advance their mathematical capabilities toward the IMO level, the scarcity of challenging, high-quality problems for training and evaluation has become a significant bottleneck. Simultaneously, recent code agents have demonstrated sophisticated skills in agentic coding and reasoning, suggesting that code execution can serve as a scalable environment for mathematical experimentation. In this paper, we investigate the potential of code agents to autonomously evolve existing math problems into more complex variations. We introduce a multi-agent framework designed to perform problem evolution while validating the solvability and increased difficulty of the generated problems. Our experiments demonstrate that, given sufficient test-time exploration, code agents can synthesize new, solvable problems that are structurally distinct from and more challenging than the originals. This work provides empirical evidence that code-driven agents can serve as a viable mechanism for synthesizing high-difficulty mathematical reasoning problems within scalable computational environments. Our data is available at https://github.com/TarferSoul/Code2Math.