Enhancing Mathematical Problem Solving in LLMs through Execution-Driven Reasoning Augmentation

AI summary

Overview

Research area: Artificial Intelligence / Large Language Model reasoning, specifically execution-guided mathematical problem solving with multi-agent and program-based methods.

Technical level: Advanced. The paper assumes familiarity with Chain-of-Thought prompting, program-aided reasoning agents, multi-agent pipelines, and benchmark evaluation for LLM math reasoning.

Scope: One sentence: The paper introduces Iteratively Improved Program Construction (IIPC), a reasoning agent that treats executable programs as a revisable representation of an LLM's reasoning chain and pairs it with a separate text-only reasoning branch to reduce over-reliance on program outputs.

What This Paper Is About

Existing LLM reasoning agents for math either commit to a fixed forward sequence of steps that cannot be revised, or rely on self-evaluation that often fails to detect and fix its own errors. Program-based agents add deterministic execution signals, but they typically generate one-off code and can be misled into over-trusting flawed or irrelevant program output.

The paper's goal is a reasoning method that keeps a manipulable, editable representation of the reasoning process, corrects it using execution feedback plus a memory of past mistakes, and still retains the base model's native token-level reasoning so that programmatic context does not dominate the final answer.

Key Contributions

1. A new reasoning method, IIPC (Iteratively Improved Program Construction). It refines programs through execution-guided feedback, integrates execution outputs into the model's own reasoning, and is designed to surpass other code-based, state-of-the-art, non-ensemble reasoning agents on difficult math benchmarks.

2. A released reasoning-trace corpus. The corpus includes problem statements, initial propositions, generated code, execution outputs, integrated deliberation, and final answers, intended to support reproducible evaluation, error analysis, and future work on program-centric reasoning.

3. A comprehensive evaluation. IIPC is compared against PoT, MACM, and CR across five LLMs (GPT-4o mini, Gemini 2.0 Flash, Mistral Small 3.2 24B, Gemma 3 27B, Llama 4 Maverick) and multiple math benchmarks, and is reported to surpass those baselines on the majority of benchmarks and models.

4. Ablations isolating architectural components. The authors separately measure iterative refinement, the persistent reflection memory, the dual-branch separation, decoding temperature, voting aggregation, and agent complexity overhead on GSM8K.

Main Findings

• IIPC leads on MATH for most models. On the MATH benchmark, IIPC reached 94.13% (Gemini 2.0 Flash), 90.83% (Mistral Small 3.2 24B), 90.56% (Gemma 3 27B), and 91.23% (Llama 4 Maverick). The PoT baseline scored 92.58%, 89.62%, 89.01%, and 88.94% respectively on those same models.

• PoT beats IIPC on GPT-4o mini. On MATH, PoT achieved 81.19% versus IIPC's 80.98% (a -0.21 point difference). The authors suggest that on models with lower reasoning capacity, IIPC's additional complexity can be counterproductive.

• IIPC leads on AIME for most models. IIPC scored 64.20% (Gemini 2.0 Flash), 52.52% (Mistral 3.2 24B), 50.48% (Gemma 3 27B), and 69.77% (Llama 4 Maverick), versus PoT at 59.16%, 48.12%, 46.20%, and 62.49%. On GPT-4o mini, PoT again led at 31.40% versus IIPC's 29.05% (-2.35).

• Baselines trail on AIME by wide margins. CR scored 23.90 / 53.48 / 40.09 / 41.69 / 62.17, and MACM scored 17.79 / 51.98 / 37.62 / 41.69 / 62.17 across the five models.

• IIPC's advantage grows with problem difficulty. On the most challenging Level-5 MATH problems (Llama 4 Maverick), IIPC achieved 80.73% accuracy, outperforming the next best method by 6.31 points.

• Domain-level gains on MATH. With Llama-4-Maverick, IIPC had the highest accuracy in every MATH domain except Counting and Probability, with the largest gains in Pre-calculus (+2.79%, 89.77%) and Intermediate Algebra (+3.25%, 87.44%).

• Ablations show each component helps on Gemini 2.0 Flash. From PoT to the full IIPC: MATH rose 92.58 → 94.13 (+1.55) via IIPC-NS-NMS 93.19 (+0.61), then IIPC-NS 93.59 (+1.01); AIME rose 59.16 → 64.20 (+5.04) via 60.77 (+1.61), then 61.52 (+2.36). Iterative refinement, reflection memory, and dual-branch separation each contributed separately.

• Components only partially compensate on GPT-4o mini. On AIME with GPT-4o mini, PoT-NC scored 31.19 and PoT 31.40, while IIPC-NS-NMS scored 26.90 (-4.50), IIPC-NS 27.12 (-4.28), and full IIPC 29.05 (-2.35) — closer to PoT, but still below it.

• On GSM8K, plain CoT is best. CoT scored 94.54 / 96.06 / 95.22 / 95.45 / 96.13 across the five models; IIPC scored 94.24 / 95.60 / 95.00 / 94.84 / 95.53. The authors attribute this to GSM8K's simpler structure saturating the benefit of iterative refinement, while noting IIPC does not significantly degrade accuracy.

• IIPC is robust to decoding temperature. On AIME with Gemini 2.0 Flash, accuracy ranged from 63.13% to 64.52% across temperatures 0.1 to 0.9, a swing of 1.39 percentage points; the best was 0.3 (64.52%).

• Voting helps IIPC least, suggesting lower single-trajectory variance. On 245 Level-5 MATH problems with Llama 4 Maverick, IIPC scored 78.78% without voting — higher than the no-voting scores of CoT (70.61), PoT (73.47), CR (68.57), and MACM (69.80). With voting, IIPC and PoT both reached 80.30%. The gains from voting were +1.52 for IIPC, versus +6.83 (PoT), +3.68 (CoT), +9.8 (CR), and +6.53 (MACM).

Methodology in Plain English

IIPC treats a program as an explicit, inspectable version of the model's reasoning chain rather than a one-off code snippet.

1. Extract propositions. The model reads the problem and writes out the key statements and facts needed to solve it.
2. Write a first program. From the problem and those propositions, the model generates a Python program that represents its reasoning. Programs are restricted to numpy, math, sympy, scipy, and scikit-spatial; they must avoid list comprehensions and recursion, use print statements, include verbose comments, and include a section that verifies the final answer.
3. Execute it. The program is run, producing output, an error, or both.
4. Branch based on the result. If execution errored, a correction component fixes only the offending code segment while preserving the reasoning chain. If execution succeeded, a process-validation component assesses the program's validity and output.
5. Remember mistakes. Every error produces a short reflection stored in a persistent memory of past mistakes, so later refinements avoid repeating failed approaches rather than resampling them. The tested configuration allows a maximum of two process validations and two error corrections after each process validation.
6. Keep a separate text branch. Independently, the model produces a pure Chain-of-Thought trace and a provisional answer from the problem and propositions alone, uncontaminated by program output.
7. Merge only at the end. The most recent working program, its output, and the CoT trace are combined through a structured integration prompt to produce the final answer.

Evaluation used a balanced subset of MATH (1483 problems, covering 35 topic-difficulty bins) and the complete AIME set (933 problems, 1983–2024), with GSM8K as a lower-complexity check. Baselines were CR, MACM, and PoT, all run without multi-trajectory voting in the main evaluation at decoding temperature 0.1; MACM was adapted to a single run, which the authors note may handicap it. Correctness was judged first by deterministic equivalence, falling back to LLM-based judging with LLaMA 4 Maverick at temperature 0. Exact matching verified 97.78% of AIME answers, 89.35% of GSM8K answers, and 72.85% of MATH answers deterministically.

Why This Matters

Impact on research. The paper reframes program generation as an editable reasoning state rather than a finalized artifact, and it proposes dual-branch separation as a way to keep execution signals from biasing token-level reasoning. It also releases error-analysis data (reasoning traces) that other researchers can use to study how program-based agents fail.

Real-world applications:

• Education technology. The paper explicitly cites education as a target domain for reliable symbolic math reasoning, with potential for step-by-step tutoring systems.
• Scientific discovery and optimization. The authors list these as application areas where reliable symbolic reasoning is essential.
• Financial modeling. Also cited by the authors as an application area.
• Tool-using assistants. The approach is a general template for agents that must combine program execution with language reasoning while avoiding over-trust in tool output.

Industry relevance. The released code, evaluations, sampled datasets, and reasoning traces support reproducibility. The paper's finding of a capability threshold — that its gains appear on higher-capacity models while simpler program-based approaches remain better on GPT-4o mini — is directly relevant to deployment decisions about which reasoning architecture to pair with which model size. The authors also flag that IIPC is token-intensive because it regenerates refined or corrected programs each iteration, a cost consideration for production use.

Future Directions

1. Improve token efficiency. The paper names token cost as an explicit limitation, since IIPC regenerates refined or corrected programs every iteration. This is framed as a target for subsequent research.

2. Adapt to the model's reasoning capacity. The paper suggests adapting IIPC to different capability levels, motivated by GPT-4o mini lagging behind PoT on both MATH and AIME.

3. Extend beyond mathematics. The authors propose applying IIPC to other domains that require verifiable, structured reasoning.

4. Open questions raised by the results. That plain CoT outperformed all agents on GSM8K raises the question of when agentic overhead pays off at all; that IIPC benefits far less from voting than baselines raises the question of whether reduced single-trajectory variance is the more valuable property to optimize. The theoretical sections on error propagation, reflection-memory regret, and dual-branch disagreement are described as conceptual abstractions rather than explicitly optimized objectives, leaving room to test them directly.

Target Audience

Researchers working on LLM reasoning, program-aided and tool-using agents, and multi-agent deliberation pipelines will get the most from this paper, particularly those interested in self-correction, execution feedback, or reasoning-state representation. It is also relevant to practitioners choosing between Chain-of-Thought, Program-of-Thoughts, and iteratively refined program agents for deployment across models of different capability levels, and to AI-in-education researchers interested in verified step-by-step mathematical solutions. The ablations and released reasoning traces make it useful for readers who want to reproduce or extend program-centric reasoning evaluations rather than only consume headline accuracy numbers.

Authors’ abstract

Mathematical problem solving is a fundamental benchmark for assessing the reasoning capabilities of artificial intelligence and a gateway to applications in education, science, and engineering where reliable symbolic reasoning is essential. Although recent advances in multi-agent LLM-based systems have enhanced their mathematical reasoning capabilities, they still lack a reliably revisable representation of the reasoning process. Existing agents either operate in rigid sequential pipelines that cannot correct earlier steps or rely on heuristic self-evaluation that can fail to identify and fix errors. In addition, programmatic context can distract language models and degrade accuracy. To address these gaps, we introduce Iteratively Improved Program Construction (IIPC), a reasoning method that iteratively refines programmatic reasoning chains and combines execution feedback with the native Chain-of-thought abilities of the base LLM to maintain high-level contextual focus. IIPC surpasses competing approaches in the majority of reasoning benchmarks on multiple base LLMs. All code and implementations are released as open source.