Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks

AI summary

Overview

Research area: Machine learning / large language model training data for code generation (arXiv:2510.23208v1 [cs.LG], presented at the 4th Deep Learning for Code Workshop, DL4C).

Technical level: Intermediate. The paper assumes familiarity with instruction tuning, LoRA/QLoRA fine-tuning, and code benchmarks such as HumanEval and MBPP, but its pipeline is described in fairly accessible stages.

Scope: The paper describes a synthetic data-generation pipeline that produces nearly 800k instruction–reasoning–code–test quadruplets and shows, through controlled fine-tuning experiments on Phi-2 2.7B and CodeGemma-2B, that reasoning-augmented and diverse data improves coding benchmark performance.

What This Paper Is About

Large language models are good at writing code, but the datasets used to train them usually pair a problem with a final solution and leave out the intermediate reasoning that connects the two. The authors argue this missing "how" signal, plus a shortage of large, diverse, and verifiably correct code datasets, limits how well models generalize to unfamiliar coding problems. Their goal is to build a scalable pipeline that generates synthetic coding tasks containing an instruction, a step-by-step reasoning trace, a working solution, and executable tests, then demonstrate that fine-tuning on that data improves coding benchmark results.

Key Contributions

1. A scalable synthetic data pipeline producing nearly 800k instruction–reasoning–code–test quadruplets, each combining a task, a reasoning trace, a working solution, and executable tests.
2. A four-part construction process: curated contest problems (e.g., LeetCode-style, Codeforces, AtCoder), web-mined content filtered by a FastText relevance classifier, reasoning-guided data expansion, and multi-stage execution-based validation.
3. Genetic-Instruct, a genetic-mutation-style algorithm that evolves new tasks through crossover and mutation while keeping reasoning traces consistent with the resulting code, increasing task diversity.
4. Empirical evidence that reasoning-aware synthetic data can substitute for model scaling, generalize across architectures, and outperform leading open-source datasets under identical sample budgets.

Main Findings

• Baseline improvements on Phi-2 2.7B: The base model scored 45.7% (Base) and 40.9% (Extra) on HumanEval, and 62.7% / 51.6% on MBPP. At 25k synthetic samples, pass rates reached 56.1% / 51.8% on HumanEval and 65.6% / 55.3% on MBPP, representing nearly +10 absolute points over baseline on HumanEval.
• Curated data helps little by comparison: Fine-tuning on the LeetCode dataset moved Phi-2 to 47.6% / 42.1% on HumanEval and 63.0% / 51.6% on MBPP, described as only marginal improvement.
• Gains scale with dataset size: At 5k synthetic samples Phi-2 reached 54.3% / 49.4% (HumanEval) and 64.3% / 54.5% (MBPP); at 10k, 54.9% / 50.6% and 65.6% / 55.3%.
• Efficiency over scale: The fine-tuned Phi-2 2.7B achieved competitive, and in some cases superior, performance compared to substantially larger models including CodeLlama-70B, Llama3-8B-instruct, and DeepSeek-Coder-33B-base.
• Generalization across architectures: CodeGemma-2B improved from 23.2% to 37.8% on HumanEval base tests (+14.6 points) and from 55.6% to 62.4% on MBPP base tests (+6.8 points) after fine-tuning on 25k samples, with its base/extra scores rising from 23.2 / 17.7 and 55.6 / 45.8 to 37.8 / 32.3 and 62.4 / 51.6.
• Diversity beats homogeneity: On matched 5k subsets, diverse data reached 56.7% versus 50.0% (homogeneous) on HumanEval, 51.8% versus 45.7% on HumanEval+, 66.1% versus 64.6% on MBPP, and 55.0% versus 53.7% on MBPP+.
• General reasoning is preserved: Across HellaSwag (Acc 55.88 / 55.22 / 55.50; Acc_Norm 73.76 / 73.00 / 73.32), WinoGrande (75.93 / 76.09 / 76.01), and MMLU (54.37 / 53.94 / 54.14) for the base, 5k-fine-tuned, and 25k-fine-tuned models, accuracy remained essentially unchanged.
• Better than other open resources at equal budget: When fine-tuned on matched 5k subsets, models trained on this dataset consistently achieved higher pass rates than those trained on EpiCoder-func-380k and Self-OSS-Instruct-SC2-Exec-Filter-50k, with the largest gains on HumanEval.
• No benchmark leakage: Hash comparison found zero overlap between the dataset and HumanEval or MBPP.

Methodology in Plain English

The pipeline starts from a small seed set of curated programming tasks — roughly 2.3k LeetCode-style problems with titles, descriptions, constraints, and examples, which support automatic test generation. Because that is too small, the authors added contest problems from Codeforces and AtCoder, combining existing HuggingFace datasets with custom scraping. To go further, they trained a FastText classifier on the curated problems and ran it over the 3B-document DCLM-Baseline corpus (a high-quality subset of Common Crawl), keeping documents at a strict 90% relevance threshold and extracting about 4M candidates.

Those raw materials are then converted by Qwen2.5-Coder-7B-Instruct into standardized quadruplets: the model rewrites each problem as a self-contained instruction, writes a step-by-step reasoning trace, and produces three candidate solution–test pairs. Candidate solutions are executed in isolated Python containers with limits on runtime, memory, and external calls; the first solution passing all its tests is kept, and samples with no passing candidate are discarded. Using Qwen2.5-Coder-7B-Instruct was a practical choice balancing efficiency and capability, and the authors note the pipeline is model-agnostic.

To grow the dataset, they use a Genetic-Instruct loop. A pool of validated instructions acts as a population: a crossover operator gives the Instructor-LLM five seed tasks as few-shot examples and asks it to merge elements from at least two into one new instruction plus a consistent reasoning trace, while a mutation operator perturbs a single task by tightening or adding constraints, increasing reasoning depth, or expanding scope. The Coder-LLM then produces three candidate solution–test pairs, executed in secure Apptainer containers, and a Judge-LLM verifies formatting, clarity, and that the code actually solves the instruction in line with the reasoning trace. The instruction pool is refreshed every 200k accepted tasks by mixing validated data back with the seed pool.

Deduplication uses MiniLM-L6-v2 embeddings with FAISS approximate nearest-neighbor search; pairs with cosine similarity above 0.90 are flagged and passed to a locally hosted Gemma-3-27B-IT model to judge functional duplication. Confirmed duplicates are merged with a union–find clustering procedure retaining one representative per class.

For evaluation, the authors fine-tuned Phi-2 (a 2.7B-parameter transformer from Microsoft) with QLoRA, setting rank r = 16, scaling factor alpha = 16, and targeting modules [q_proj, v_proj, k_proj, dense] for 10 epochs. Benchmarks were HumanEval and MBPP evaluated through the EvalPlus framework. Cross-model experiments used CodeGemma-2B with a standardized QLoRA setup chosen via a one-epoch configuration search over eight variants (rank r in {16, 32}, scaling alpha = 2r, target modules [q_proj, v_proj] or [q_proj, v_proj, k_proj], and save_head in {True, False}). All experiments ran on a single NVIDIA A100 GPU (80GB): configuration search runs took roughly 1–2 hours each, and 10-epoch fine-tuning on the largest 25k-example dataset took up to roughly 12 hours.

Why This Matters

Impact on research: The paper argues that the key to better code models is not raw dataset size but the integration of reasoning and diversity. It provides evidence that reasoning-aware synthetic data can substitute for model scaling, and it releases the dataset and generation pipeline (with a GitHub repository and Hugging Face dataset) to support reproducibility and further work.

Real-world applications:

• Training smaller, cheaper code assistants that perform competitively without access to tens-of-billions-parameter models or large-scale distributed infrastructure.
• Improving model behavior on harder, multi-step programming problems: gains were most pronounced on HumanEval, whose tasks often require multi-step reasoning.
• Producing interpretable training signals, since each sample includes a reasoning trace that explains how a solution is derived, which supports debugging and oversight of generated code.
• Building data pipelines for regulated or resource-constrained settings where collecting human-annotated reasoning data is prohibitively expensive.

Industry relevance: Scaling to tens of billions of parameters requires specialized hardware, large-scale distributed training, and high inference costs, all of which limit accessibility. The authors frame carefully curated reasoning-focused data as a more compute-efficient and sustainable path, lowering both training and deployment costs and broadening access to competitive code generation for researchers and practitioners with modest resources.

Future Directions

• Extending the pipeline to other programming languages: a stated current limitation is that the pipeline supports only Python, and broadening it will require adapting the execution-based validation framework to ensure functional correctness in more diverse coding environments.
• Multilingual programming targets explicitly named as next steps include Java, C++, and JavaScript.
• Deploying the pipeline at pretraining scale rather than only for fine-tuning.
• Further isolating how dataset structure (diversity versus homogeneity) and reasoning-trace quality interact with downstream performance, given that diversity proved more valuable than raw sample count at small and medium budgets.

Target Audience

Researchers and practitioners working on code generation with LLMs, particularly those interested in synthetic data generation, instruction tuning, and efficient fine-tuning. It is also relevant to engineers who need strong code models under limited compute budgets, and to dataset builders looking for a template that combines classifier-based corpus filtering, execution-based validation, evolutionary task expansion, and multi-stage deduplication. Readers should be comfortable with fine-tuning terminology such as QLoRA, rank, and target modules, and with benchmark names like HumanEval, MBPP, and EvalPlus.

Authors’ abstract

Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and well-aligned with human reasoning. Most existing resources pair problems with solutions, but omit the intermediate thought process that guides coding. To close this gap, we present a scalable synthetic data generation pipeline that produces nearly 800k instruction-reasoning-code-test quadruplets. Each sample combines a task, a step-by-step reasoning trace, a working solution, and executable tests, enabling models to learn not just the what but also the how of problem solving. Our pipeline combines four key components: curated contest problems, web-mined content filtered by relevance classifiers, data expansion guided by reasoning patterns, and multi-stage execution-based validation. A genetic mutation algorithm further increases task diversity while maintaining consistency between reasoning traces and code implementations. Our key finding is that fine-tuning LLMs on this dataset yields consistent improvements on coding benchmarks. Beyond raw accuracy, reasoning-aware data can substitute for model scaling, generalize across architectures, and outperform leading open-source alternatives under identical sample budgets. Our work establishes reasoning-centered synthetic data generation as an efficient approach for advancing coding capabilities in LLMs. We publish our dataset and generation pipeline to facilitate further research.