Chain of Execution Supervision Promotes General Reasoning in Large Language Models

AI summary

Overview

Research area: Machine learning / large language model training data and reasoning supervision; specifically, code-derived training corpora for improving general reasoning.

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought prompting, continued pretraining vs. instruction tuning, and benchmark evaluation, but the core idea is accessible to anyone who understands supervised finetuning.

Scope: The paper introduces TracePile, a 2.6-million-sample corpus of natural-language, step-by-step code execution traces ("Chain of Execution"), and demonstrates across four base models and 20 benchmarks that training on these traces improves mathematical, code, logical, and algorithmic reasoning.

What This Paper Is About

Code contains many reusable reasoning patterns — divide-and-conquer, topological ordering, backtracking, enumeration — but those patterns are buried inside syntax, boilerplate, and implementation noise, so training on raw code is a poor way to teach general reasoning. The authors convert each step of a program's execution into an explicit, natural-language chain-of-thought narration, then test whether supervising models on these "Chain of Execution" traces improves reasoning not just on code, but on math, logic, and algorithm tasks as well.

Key Contributions

1. TracePile, a 2.6M-sample / ~19B-token CoE corpus. A large-scale dataset that turns code execution into explicit step-by-step natural-language reasoning traces, drawn from three sources: Codeforces algorithmic competition submissions, 30 classical algorithms (CLRS-style, including graph tasks), and mathematical solver code from OpenMath.

2. Two reasoning-aware augmentation strategies. Query diversification generates fine-grained questions about internal execution state (e.g., "what are the stack contents after this backtracking step?"), while code diversification rewrites implementations (e.g., recursive DFS into iterative DFS) to preserve semantics but vary structure.

3. A verified generation and filtering pipeline. CoE traces are produced by Qwen-2.5-72B-Instruct with few-shot exemplars, then validated by comparing structured JSON intermediate states against ground-truth execution traces from code instrumentation or symbolic generators; incorrect samples are discarded and outputs are capped at 8k tokens.

4. Systematic evaluation across three training regimes and four base models. Continue-pretraining, instruction tuning after pretraining, and two-stage finetuning are tested on LLaMA 3-8B, LLaMA 3.1-8B, Qwen-2.5-7B, and Qwen-2.5 Coder-7B across 20 benchmarks in math, code, logic, and algorithms.

Main Findings

• Broad, consistent gains under continue-pretraining. TracePile improves every evaluated model. LLaMA-3-8B jumps from 54.8% to 63.4% average across nine math benchmarks (+8.4 points), and LLaMA-3.1-8B gains roughly 7.1% on average across the same set.

• Large gains on code-execution and algorithm tasks. LiveCodeBench-Output rises from 2.0 to 30.5 for LLaMA-3-8B and from 1.6 to 11.8 for LLaMA-3.1-8B. GraphWiz goes from 4.5 to 46.0 and GraphInstruct from 35.2 to 73.5 for LLaMA-3-8B. RuleTaker improves from 2.8 to 42.4.

• Two-stage finetuning preserves and extends gains. After TracePile followed by Tulu3-SFT, LLaMA-3-8B improves from 55.3% to 61.7% on math, and Qwen-2.5-7B from 73.6% to 75.4%. CRUX improves from 33.5 to 49.9 (LLaMA-3.1-8B) and CLRS from 17.2 to 24.5 (LLaMA-3-8B).

• Out-of-domain transfer occurs. Gains appear on benchmarks with little or no overlap with TracePile sources, including LiveCodeBench, CRUX, Zebra Logic, and RuleTaker, suggesting the supervision teaches transferable procedural reasoning rather than domain-specific memorization.

• CoE supervision beats final-answer and solution-style supervision. In a controlled comparison on Qwen-2.5-Coder-7B, TracePile (80.8 GSM8K, 50.8 MATH, 35.5 LiveCodeBench-Output, 10.8 Zebra Logic) outperforms OpenMathInstruct-1, OpenMathInstruct-2, WebInstruct, CodeI/O, and CodeI/O++ — with the advantage largest on out-of-domain tasks such as LiveCodeBench-Output, where baselines fall below 21%.

• Every data component matters. Ablations on LLaMA-3.1-8B show that removing math, algorithm, or competition data degrades performance across all four domains, not just the removed one. Removing code rewrites hurts most in code (−3.0%) and algorithm (−3.1%) reasoning, indicating cross-domain transfer and the value of structural diversity.

• Performance scales with CoE data volume, with caveats. Expanding TracePile from 50K to 4.3M samples improves all four reasoning domains. However, adding rejection sampling (TracePile++) helps math but plateaus or slightly regresses code, logic, and algorithms — suggesting rejection filtering introduces distributional bias.

• Fine-grained reasoning skills improve. On BBH subsets, TracePile boosts Tracking Shuffled Objects from 68.4 to 87.6 and Multi-Step Arithmetic from 72.0 to 82.0, indicating better state tracking and multi-hop computation.

Methodology in Plain English

The researchers started by gathering code from three places: competitive programming submissions (Codeforces), textbook classical algorithms (sorting, graph traversal, dynamic programming, and so on), and math problem solvers. Because raw competition code is messy and hardcoded to specific inputs, they used a large model (Qwen-2.5-72B-Instruct) to extract clean functions together with input-output pairs, then executed the code to verify the pairs actually matched.

To diversify the data, they generated extra questions about a program's internal state mid-execution — not just "what does this return?" but "what does the stack look like after this step?" — and had a model rewrite code into semantically equivalent but structurally different forms. They applied these strategies to algorithms and graph problems but not to math code, which typically has one clear objective.

The core step was converting execution into narration. Given a question and code, the model was shown two human-written examples and asked to produce a step-by-step natural-language account of what happens as the code runs. For tasks involving variable tracking, the model also had to output structured JSON with intermediate states, which the researchers checked against ground-truth traces from instrumented execution. Failing samples were thrown out, and any trace longer than 8,000 tokens was discarded to keep training manageable. Each rationale was sampled five times to improve the odds of getting a correct trace on hard problems.

Finally, they evaluated three training recipes: continued pretraining on TracePile alone, continued pretraining followed by general instruction tuning, and a two-stage finetune (TracePile first, then general instructions). All training used 16 H800 GPUs, batch size 512, sequence length 8192, and 3 epochs, with Tulu3-SFT as the general instruction dataset.

Why This Matters

Impact on research. The paper argues that what matters for reasoning supervision is not just correct final answers but explicit intermediate process. By showing that execution traces transfer to non-code tasks, it supports the view that procedural, step-by-step data is a broadly useful intermediate training stage — and it provides a large, reusable corpus plus a validation pipeline others can build on.

Real-world applications:

• Code assistants and IDE tools. Models trained on execution traces are better at predicting what code will do, which helps with debugging, test generation, and explaining unfamiliar code to developers.

• Mathematical and quantitative tutoring. Improvements on GSM8K, MATH, and state-tracking benchmarks suggest better step-by-step math explanations for educational products.

• Automated program verification and analysis. Stronger execution simulation supports static analysis, fuzzing, and correctness checking in software pipelines.

• Structured decision-making agents. State tracking and multi-step deduction are the same skills needed for planning, scheduling, and constraint-satisfaction agents used in operations and logistics.

Industry relevance. The gains on Codeforces-style data and LiveCodeBench matter directly to companies training code models, since a single intermediate training stage improves both code and general reasoning without requiring domain-specific labeled data at scale. The finding that rejection sampling can hurt generalization is also practically useful: more aggressive data filtering is not automatically better.

Future Directions

• Extending beyond code-grounded reasoning. The authors note TracePile is skewed toward math and algorithms. Expanding CoE-style supervision to law, planning, causal inference, and commonsense domains is the natural next step.

• Reducing dependence on frontier models. Generation relies on Qwen-2.5-72B-Instruct, which is costly and injects the generator's biases. More efficient or self-supervised trace generation is an open problem.

• Handling long reasoning traces. The 8k-token cutoff discards combinatorially long execution traces, especially for large graphs. Better long-context or compressed-trace methods could recover that signal.

• Understanding the rejection-sampling regression. Since TracePile++ improves math but hurts code and logic, the mechanism by which filtering biases the distribution needs explanation before scaling further.

• Integrating with reinforcement learning. The paper uses supervised finetuning only; combining CoE data with execution-feedback reward signals is an untested direction.

Target Audience

This paper is most valuable to LLM training researchers and engineers working on reasoning data curation, especially those building code or math models. It also benefits practitioners who need a practical intermediate training stage that improves reasoning without hand-labeled domain data, and researchers studying chain-of-thought supervision who want evidence about why process-level rather than answer-level supervision generalizes. Readers should be comfortable with finetuning terminology and benchmark evaluation to get the most out of it.

Authors’ abstract

Building robust and general reasoning ability is a central goal in the development of large language models (LLMs). Recent efforts increasingly turn to code as a rich training source, given its inherent logical structure and diverse reasoning paradigms such as divide-and-conquer, topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syntactic or implementation noise, making direct training on raw code suboptimal.To address this, we introduce TracePile, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought-style rationales, which we call Chain of Execution (CoE). The corpus spans domains including mathematics, classical algorithms and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity. We evaluate TracePile using three training setups: continue-pretraining, instruction tuning after pretraining, and two-stage finetuning. Experiments across four base models (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, TracePile boosts LLaMA3.1-8B by 7.1\% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and MMLU under two-stage fine-tuning.