Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training

AI summary

Overview

Research area: Natural Language Processing, specifically supervised fine-tuning (SFT) data design for reasoning in language models.

Technical level: Intermediate. The paper is readable for anyone familiar with language model training, but it assumes familiarity with objectives such as completion-supervised fine-tuning, negative log-likelihood, and reinforcement learning with verifiable rewards.

Scope: The paper introduces and empirically evaluates Reasoning Core, a collection of 50 procedural reasoning-task generators, and compares it against three other procedural collections under a matched completion-supervised training protocol.

What This Paper Is About

Procedural generators can produce verifiable reasoning problems at scale, but most prior work uses them as environments for reinforcement learning rather than as data for supervised fine-tuning. The authors ask a data-design question: how should broad collections of verifiable reasoning examples be designed and represented so they actually improve a model when used as supervised training data? They build Reasoning Core to answer this, and then test whether its design choices translate into better downstream performance than existing procedural collections.

Key Contributions

1. A broad, reproducible resource. Reasoning Core provides 50 generators with semantic scorers, difficulty controls, and a single interface for supervised fine-tuning, evaluation, and reinforcement learning, organized into nine descriptive groups (mathematics; formal proof and symbolic manipulation; logical and probabilistic inference; planning and games; state and reference tracking; graphs, constraints, and relations; formal languages and transduction; sets and structured data; code). The release includes generated datasets, versioned manifests, caches, behavior hashes, task-level measurements, and audit material.

2. A controlled collection comparison. The authors compare four procedural collections (Reasoning Core, Procedural Warmup, Reasoning Gym, SynLogic) across four base-model settings and several training durations, with complete held-out evaluations.

3. A repository-scale semantic audit. A procedure combining model-assisted review, API-based sampling, manual adjudication, and regression testing was applied throughout development and then retrospectively to Reasoning Gym and SynLogic, exposing subtle generator, target, and scorer failures.

4. Design findings on target representation. The paper reports that compact canonical answer targets outperformed semantically correct step-by-step rationale traces, and that semantic validity of a task does not by itself guarantee training utility.

Main Findings

• Primary 3B comparison favors Reasoning Core. At SmolLM3-3B-Base after 2,400 updates across 5 seeds, Reasoning Core achieved the highest mean scores among the compared collections on DROP F1 (41.7 ± 0.8 vs. 33.1 ± 1.0 main-only, 39.2 ± 2.4 Reasoning Gym, 32.5 ± 0.4 SynLogic, 31.3 ± 0.5 Procedural Warmup), LogiQA accuracy (47.8 ± 0.7), and ARC-Challenge accuracy (51.3 ± 0.5). It also scored highest on BBH-test (45.3 ± 1.1 vs. 43.6 ± 1.1 main-only). Reasoning Gym scored highest on MMLU-other (44.2 ± 0.3 vs. 43.6 ± 0.5 Reasoning Core).

• Every procedural collection transfers positively somewhere. Across models and durations, each of the four procedural collections yields positive reasoning transfer in at least some settings, but their relative ordering varies across metrics, model families, and training durations.

• Narrowing the task mixture hurts aggregate transfer. A 50-task Reasoning Gym subset selected by BBH-development NLL improved LogiQA but degraded most other metrics relative to the full collection.

• Semantic validity does not ensure training utility. Task-level analyses show that semantic validity alone does not predict usefulness, and the authors highlight compact targets and calibrated difficulty as important design factors.

• Step-by-step rationales underperformed compact answers. For tasks including parsing (Earley chart parser traces) and graph pathfinding (BFS and Dijkstra traces, including frontier expansion, distance updates, and predecessor choices), semantically correct rationales consistently performed worse than compact-answer supervision on both BBH development and FineWeb NLL. The ordering held with matched training budgets, without budget matching, and after making traces shorter and more token-efficient.

• Difficulty regimes matter within Reasoning Gym. After 300 updates, SmolLM2-360M reached at least .90 native reward on 6 of 95 tasks (including graph_color and propositional_logic), while 45 remained at or below .05 and count_primes stayed at zero. Intermediate final rewards showed the strongest held-out transfer; fully saturated tasks did not transfer better.

• Grid and board tasks form a negative cluster. Their apparent penalty on FineWeb was mostly explained by prompt and answer length. Long-input decisions could still help BBH while hurting retention, whereas full-state reconstruction tended to hurt both. Some constraint grids, including sudoku and futoshiki, remained weak on BBH after controlling for length.

• Token matching isolates length from dose. Across 18 paired OLMo-1B tasks, token matching improved BBH in every case but left the relationship between example length and FineWeb retention unchanged.

• Within-family task rankings are more reproducible than cross-family ones. Kendall's τ_b within the SmolLM2 family was +0.68 for Reasoning Core (84% of pairwise orderings preserved), +0.75 for Reasoning Gym (88%), and +0.88 for SynLogic (94%). Cross-family agreement was much lower: 135M–OLMo gave +0.34 (67%), +0.44 (72%), and −0.24 (35%) respectively; 360M–OLMo gave +0.38 (69%), +0.39 (70%), and −0.26 (34%).

• External audits found defects. Under conservative criteria, the audit confirmed material default-path defects in 13 of 105 Reasoning Gym tasks and nine native SynLogic generators, including a Reasoning Gym scorer that awards full credit to any nonempty answer and a SynLogic generator whose displayed constraints can contradict its stored solution. The authors stress the comparison is asymmetric and not a quality ranking.

• Zero-shot headroom remains. DeepSeek V4 Flash (roughly 300B parameters) in High reasoning mode remained below 0.75 reward on 18 of 50 evaluated tasks and below 0.90 on 32 of 50. Among the 18 tasks where Flash High reached at least 0.90, Flash Instant and Pro Instant achieved mean rewards of 0.61 and 0.62 respectively, so neither the roughly 300B Flash nor the roughly 1.6T Pro configuration fully solves the collection zero-shot.

• Verifier-backed RL is viable. In a matched single-seed GRPO run on Qwen2.5-3B-Instruct with a 200k-example budget, at the final checkpoint Reasoning Core reached 38.6% versus 23.1% on BBH-test and 62.5% versus 43.2% on MMLU, compared with Reasoning Gym. The authors note the Reasoning Gym run degraded MMLU below the base model and treat the gap as illustrative only.

Methodology in Plain English

The authors built 50 generators that each produce a problem instance, a rendered prompt, a structured reference answer, and a scoring function that checks answers semantically. Rather than using fixed difficulty presets, a global difficulty knob maps a scalar level to task parameters such as derivation depth, plan length, variable count, or branching factor. Prompts use zero-shot instructions without few-shot demonstrations, and answers are deliberately compact (a number, label, set, expression, proof-line index, plan, or short program), with deterministic canonical completions specified where multiple valid serializations exist. Scoring remains broader than the supervised target: sets are compared order-invariantly, lambda terms up to variable renaming, plans are executed, proof candidates compiled, and generated programs tested.

For the training comparison, all four collections were converted into prompt–answer examples and trained with the same completion-supervised objective. Four pretrained base models were used (SmolLM2-135M, SmolLM2-360M, OLMo-1B, SmolLM3-3B-Base), with a main-data stream blending FineWeb-Edu text equally with DOLCI instruction and conversation data. The procedural collection replaced 20% of prompt-plus-answer tokens while total updates stayed fixed; this ratio was validated on Reasoning Gym with a sweep over 10%, 20%, and 40% using BBH validation NLL at 2,400 steps. Maximum sequence length was 1,024, seeds were paired with a shared data order, and durations from 300 to 2,400 updates were evaluated. The primary measure is the relative reduction in held-out answer negative log-likelihood from adding an auxiliary collection, reported alongside accuracy and F1 at 3B where scores become discriminative.

BBH was split by the authors' own task taxonomy: non-algorithmic NLP tasks formed the development set and algorithmic tasks were held out for testing. No Reasoning Core generator instantiates, adapts, or reproduces a BBH-development task, and task-development decisions were based on 300-step runs of SmolLM2-360M and OLMo-1B only. For auditing, repeated model-assisted reviews (GPT-5.5 High and later GPT-5.6 High) examined code, prompts, targets, and scorers, while separate API passes sampled rendered examples with Claude Haiku 4.5 without reasoning at difficulty levels 0 and 1; cases receiving a reward other than 1 were flagged for adjudication using Claude Opus 4.6 and later Claude Opus 4.7, followed by human review. Model judgments were never treated as ground truth.

Why This Matters

Impact on research. The paper argues that the supervised data layer deserves direct study, citing work questioning how much reasoning reinforcement learning creates beyond capabilities established before RL. It shows that procedural collections developed for RL can serve as useful completion-supervised data, and conversely that Reasoning Core supports verifier-backed RL — but that orienting design explicitly toward completion objectives yields better overall SFT results. It also challenges the assumption that procedurally generated data is correct by construction.

Real-world applications:

• Building supervised fine-tuning datasets for reasoning where generated examples need verifiable, automatically checked answers.
• Constructing evaluation suites with calibrated difficulty for models of different sizes.
• Providing verifier-backed reward signals for reinforcement learning on reasoning tasks.
• Auditing existing synthetic-data pipelines for mismatches between generation, rendering, targets, and scoring.

Industry relevance. Teams that generate synthetic reasoning data at scale can reuse the design principles here — compact canonical targets, difficulty controls separated from rendering, and semantic scorers distinct from supervised targets — and can adopt the layered audit procedure. The released material (roughly 10B generated tokens under permissive licenses, plus diagnostics) makes these conclusions inspectable at the generator level.

Future Directions

• Scaling beyond 3B. The authors state their evidence stops at 3B parameters, so whether these trends persist at larger scales remains unknown, and model-family coverage is limited.

• Sequential SFT then RL. Completion-supervised training and verifier-backed RL were evaluated as separate interventions. It is untested whether procedural SFT changes subsequent RL, or whether reusing the same collection across both stages is beneficial.

• Broader evaluation domains. The evaluation emphasizes formal and closed-answer reasoning and does not establish improvements on open-ended, multimodal, or real-world agentic tasks. The authors also do not compare against matched mixtures of DOLCI's individual human-curated components such as GSM8K or MATH, nor evaluate formal-logic benchmarks beyond the MMLU and BBH scores used.

• Rethinking rationale targets. The negative result on step-by-step traces raises the question of when, if ever, faithful solver transcripts are a better training target than the answers they compute.

Target Audience

Researchers and engineers working on synthetic data generation, supervised fine-tuning, and reasoning training for language models. It is most useful to practitioners who build procedural or verifiable task collections and need guidance on target representation, difficulty calibration, and semantic auditing, as well as to researchers studying how pre-RL supervised data shapes later reinforcement learning.

Authors’ abstract

Procedural generators produce useful verifiable reasoning problems at scale, but have received less attention as data for completion-supervised fine-tuning. We introduce Reasoning Core, a collection of 50 generators spanning mathematics, logic, planning, state tracking, formal languages, structured data, games, causality, and code, with semantic scorers, difficulty controls, and task evaluators. Under a matched completion-supervised protocol, we compare Reasoning Core with Procedural Warmup, Reasoning Gym, and SynLogic across four base-model settings and multiple training durations. In the primary 3B comparison, Reasoning Core achieves the highest mean scores on DROP, LogiQA, and ARC-Challenge, exceeding both the baseline without procedural data and all three alternative procedural collections. Task-level analyses show that semantic validity alone does not ensure training utility, highlighting compact targets and calibrated difficulty as important design factors. We ran audits combining model-assisted review, human adjudication, and regression testing. Applied throughout Reasoning Core development and to the other collections, they reveal subtle mismatches among generation, rendering, targets, and scoring, a reminder that procedural generation alone does not guarantee correctness. The library, generated datasets, and audit material are publicly available.