More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning

AI summary

Overview

• Research area: Natural Language Processing — specifically data curation (selection and synthesis) for training large language models on mathematical reasoning.
• Technical level: Intermediate. Readers need basic familiarity with LLM training stages (pretraining, supervised fine-tuning) and standard math reasoning benchmarks, but no deep mathematical background.
• Scope: A systematic, industrial-pipeline evaluation of open-source math datasets and data synthesis methods, testing whether adding more data or curating better data produces stronger mathematical reasoning models (arXiv:2510.07169v1, cs.CL, 08 Oct 2025).

What This Paper Is About

Many papers propose new ways to build training data for LLM reasoning, but most are evaluated under academic conditions rather than the messy, mixed-workload reality of industrial model development. This paper asks a practical question: when a team already has a large validated training mixture, does adding a new dataset actually help, and if so, which kinds of data construction work? The authors test a wide range of open-source datasets and synthesis techniques under one shared evaluation pipeline, including several attempts that failed, and distill concrete rules for choosing data.

Key Contributions

1. A unified evaluation pipeline that mirrors training and deployment. The authors use DeepSeek-V2-Lite or Qwen2.5-3B as base models, establish a baseline on a large validated dataset mixing code and math data across pretraining and SFT stages, then anneal in each new dataset at a weight of 0.2 (20% new data, 80% validated data), following the annealing method of Grattafiori et al. (2024) and Gu et al. (2024). A dataset counts as effective only if it beats the baseline.
2. A systematic evaluation of open-source datasets and construction methods. Web-scraped corpora (IndustryCorpus2 / IC2-Math, opc-fineweb-math-corpus), small high-quality sets (LIMO, s1K), a large principled set (OpenR1-Math-220K), distilled data (NaturalReasoning-QwQ), weakness-guided retrieval and augmentation datasets, textbook-derived question-answer datasets, and pretraining refinement (Math-Cosmo) are all measured against the same baseline.
3. Distilled practical data-selection strategies. The paper converts its experiments into rules: be cautious with data aggregated from diverse web sources, prioritize datasets distilled by advanced reasoning models (e.g., DeepSeek-R1, QwQ-32B), and scale generation only while preserving quality.
4. A set of promising future directions, including RL-inspired data synthesis. Proposed ideas include multiple solution paths per problem, reward-like signals, synthetic data with limited noise, hierarchical data training, integration of data from diverse models, better data mixing, and curriculum learning.

Main Findings

• More is not automatically better. Datasets built by large-scale web scraping were unhelpful and sometimes harmful. IC2-Math scored 36.74 on MATH (Δ −3.7 versus baseline) and showed a BBH drop of −6.28; opc-fineweb-math scored 32.9 on MATH (Δ −0.14) with Sanitized-MBPP at −3.89. The paper attributes this to noise that simple cleaning cannot remove.
• Small high-quality data yields modest gains, but limited volume caps the benefit. LIMO reached 54.9 on MATH (Δ −0.28) and 86.58 on GSM8K (Δ −0.15); s1K reached 54.52 on MATH (Δ −0.66) and 89.01 on GSM8K (Δ +2.28). Gains are described as modest because of limited data volume.
• Principled large-scale generation works best among the tested datasets. OpenR1-Math-220K, with over 220,000 samples, produced +8.96 accuracy on MATH (score 52.2), plus GSM8K Δ +0.91, at some cost to coding (OpenAI-Humaneval Δ −3.05, Sanitized-MBPP Δ −3.11).
• Reasoning-model distillation clearly beats the original data. Starting from NaturalReasoning questions, regenerating answers with QwQ-32B and filtering inconsistent answers produced NaturalReasoning-QwQ, which gained +1.92 on MATH (49.68), +1.47 on MathBench-a, +1.39 on MMLU-Pro, and +1.16 on BBH.
• Retrieval alone is weak; augmentation helps. Using MathBERT embeddings and FAISS, the authors retrieved the top 20 most semantically similar examples per failure case from the MATH dataset, collecting over 75,000 examples and more than 82 million tokens (math-retrieval). It gave little improvement, especially on mathematics (MATH Δ −1.7), whereas the augmented version reached MATH 51.48 (Δ +3.72, math-retrieval-augmented), and the weakness variant scored 48.24 (Δ +0.48).
• Data presentation matters in pretraining. Collecting intermediate algebra books yielded 192 samples and roughly 0.13 billion tokens with little benchmark improvement. Rewriting the corpus into a more educational, interpretable format with Qwen2.5-72B produced Math-Cosmo (760 million tokens), which improved MATH by +1.72 (49.48) and MathBench-a by +1.33. For comparison, Cosmopedia, the approach replicated, generated over 30 million files and 25 billion tokens with Mixtral-8x7B-Instruct-v0.1.
• Identical construction methods can produce very different outcomes. Two textbook-based question-answer datasets built with the same chapter/section segmentation, LLM extraction, LLM-based quality filtering, and MinHash deduplication behaved differently: Calculus-QA gained +4.66 on MATH, +2.04 on GSM8K and +5.05 on GPQA-Diamond but lost −5.87 on MathBench-t, while Intermediate Algebra-QA showed small or negative changes on those benchmarks. The authors link this to the data mixing strategy.
• Rule-based and length-based complexity do not deliver. Rule-based generation (modus ponens-style logical inference, Knights and Knaves puzzles, graph-based dependency math word problems) gave limited reasoning gains, especially in mathematics. Combining Long CoT reasoning chains from reasoning LLMs with distilled data also gave limited improvement, and categorization by reasoning-step length showed no clear correlation with reasoning performance.
• Overall verdict. Table 5 contrasts "More Data" (noisy web-crawled sources, broad aggregation without filtering, purely rule-based synthesis, difficulty partitioned by reasoning chain length, direct integration) with "Better Data" (structured interpretable sources, multi-stage filtering and model-based selection, distillation from advanced models plus weakness-guided generation, fine-grained difficulty hierarchy with curriculum learning, balanced interference-aware mixing); the latter produces consistent improvements and cost-efficiency.

Methodology in Plain English

The authors build one fixed test harness instead of comparing each technique under its own settings. A small base model (DeepSeek-V2-Lite or Qwen2.5-3B) is first trained on a large validated dataset of code and mathematics covering both pretraining and SFT. That trained model is the baseline.

Then, for each candidate dataset, the model is trained again on a mixture where 20% of samples come from the new dataset and 80% from the validated dataset — an "annealing" setup borrowed from prior work and chosen because it resembles how production teams slot new data into an existing recipe. Fourteen benchmarks spanning knowledge (MMLU, MMLU-Pro, CMMLU, GPQA-Diamond), reasoning (HellaSwag, BBH, DROP), math (MATH, GSM8K, MathBench-a, MathBench-t) and coding (OpenAI-HumanEval, Sanitized-MBPP) are scored, using accuracy except for coding, where Pass@1 is used. Math improvement is the target, but regressions elsewhere are tracked.

For data synthesis, the authors replicate existing recipes: Cosmopedia-style rewriting of an OCR-extracted intermediate algebra book corpus into educational prose with Qwen2.5-72B; Dolphin-R1-style answer generation with QwQ-32B on NaturalReasoning followed by consistency filtering; weakness-guided retrieval from MATH failures using MathBERT embeddings and FAISS; weakness-guided generation with a two-round consistency check; and textbook question-answer extraction with LLM quality filtering plus MinHash deduplication. They also document attempts that did not work.

Why This Matters

This paper addresses a gap between academic data-construction papers and industrial practice, where models handle many tasks at once and heterogeneous data can create conflicting objectives and degraded performance. Its central message — that structuring data in more interpretable formats, or distilling from stronger models, often outweighs scaling up data volume — gives teams a concrete basis for spending curation effort rather than compute.

Real-world applications:

• Math and STEM tutoring systems that need reliable step-by-step reasoning, benefiting from the observation that interpretable, well-structured explanations help more than raw volume.
• Code-plus-reasoning assistants, since the pipeline evaluates code and math jointly and reports cross-domain regressions such as the coding drops seen with OpenR1-Math-220K.
• Cost-conscious model training pipelines, where the baseline-versus-annealed comparison at a 0.2 weight is a directly reusable screening test for whether to buy into a new dataset.
• Data marketplace and dataset procurement decisions, where the paper's warnings about web-scraped corpora and its contrast between LIMO/s1K-scale sets and 220,000-sample sets inform what is worth licensing or building.

Industry relevance is explicit: the evaluation protocol is described as matching practices commonly adopted in production environments for robustness and reproducibility, and the paper reports unsuccessful attempts alongside successful ones so that practitioners do not repeat them.

Future Directions

• RL-inspired data synthesis, presented as exploratory since no experimental results are given: generating multiple solution paths per problem (including incorrect ones), attaching reward-like signals such as quality metadata or annotations of reasoning flaws, including nearly-correct solutions with error explanations, presenting data in increasing difficulty order, and combining synthetic data from models of different sizes or domain expertise.
• Data mixing strategy. Indiscriminate mixing is reported to reduce the benefit each dataset brings, suggesting interference effects, and the striking divergence between two identically constructed textbook datasets points to mixing as an open problem needing dedicated study.
• Curriculum learning done properly. Since partitioning difficulty by reasoning-step count showed no clear correlation with performance, the authors call for more fine-grained, task-relevant difficulty hierarchies and better ways to embed curriculum principles in training.
• Broader validation. The limitations section asks for study across broader tasks and larger model scales, since the base models used are relatively small and the analysis is confined to mathematical reasoning; it also notes that statistical validation such as confidence intervals and p-values is hard to run at this scale, so case studies were used instead.

Target Audience

This paper is most useful to applied researchers and engineers who build and curate training data for LLMs — particularly those working on mathematical reasoning, STEM domains, or mixed-task industrial pipelines. It also suits data strategists deciding which datasets to acquire or synthesize, and academic researchers who want a realistic yardstick for how their proposed data methods perform outside controlled benchmarks. Readers looking for a novel model architecture or a new training algorithm will not find one here; the contribution is empirical guidance and evaluation methodology.

Authors’ abstract

The reasoning capabilities of Large Language Models (LLMs) play a critical role in many downstream tasks, yet depend strongly on the quality of training data. Despite various proposed data construction methods, their practical utility in real-world pipelines remains underexplored. In this work, we conduct a comprehensive analysis of open-source datasets and data synthesis techniques for mathematical reasoning, evaluating them under a unified pipeline designed to mirror training and deployment scenarios. We further distill effective data selection strategies and identify practical methods suitable for industrial applications. Our findings highlight that structuring data in more interpretable formats, or distilling from stronger models often outweighs simply scaling up data volume. This study provides actionable guidance for integrating training data to enhance LLM capabilities, supporting both cost-effective data curation and scalable model enhancement. We hope this work will inspire further research on how to balance "more data" versus "better data" for real-world reasoning tasks.