Selecting Diverse SFT Traces Improves Post-RL Generalization

AI summary

Overview

Research area: Machine learning / large language model post-training, specifically data selection for supervised fine-tuning (SFT) that precedes reinforcement learning with verifiable rewards (RLVR).

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with SFT, group-relative RL (GRPO), pass@k metrics, and embedding-style representations.

Scope: A controlled empirical study showing that selecting reasoning traces with diverse reasoning routes at a fixed SFT budget improves post-RL generalization, plus a cheap CPU-only selection method validated on three released corpora.

What This Paper Is About

Post-training pipelines typically verify a large pool of candidate solutions, keep a subset for SFT, then run RL on the model's own attempts. The paper asks whether which verified solutions are kept matters — not just how many, and not just whether they are correct. Its central claim is that routes (the sequences of reasoning steps) that differ from one another prepare a model better for RL than routes that repeat the same procedure, even when the total amount of training data, the student, the recipes, and the checkpoints are all matched.

Key Contributions

1. A controlled study of route diversity across teacher-source sweeps and direct route selection. Every comparison matches the student model, prompt pool, SFT trajectory budget, group-relative RL recipe, and evaluation checkpoint; conditions differ only in which solutions the SFT data contains.

2. A lightweight, rule-based "topology fingerprint" for routes. Each verified solution is parsed into steps, labelled from a dataset-specific vocabulary, and summarized as a fixed-length vector over step frequencies, transitions, tree shape, strategy patterns, dense statistics, and hashed route labels. It uses no model calls, no new generation, and no gradients, and runs on CPUs over candidate pools exceeding two million solutions.

3. A selection procedure built on that fingerprint. Diverse selection clusters fingerprints, gives each cluster a size-proportional budget, and repeatedly adds the candidate farthest from those already chosen (a coreset construction); the matched "similar" control uses nearest-centroid selection to build a set from one dense region.

4. Evidence that the benefit does not require multiple teachers. In a single-model condition where one model writes every candidate, route-diverse selection from that one pool still leads after the same RL.

Main Findings

• Post-RL coverage improves on RLVE. On RLVE, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 percentage points on environments held out from SFT, even though both conditions subsequently receive RL on those environments. The margin is positive on both SFT-seen and SFT-unseen environments and increases over the reported sampling budgets.

• The solved sets expand rather than merely trade successes. The diverse model solves 1,133 questions the similar model misses, versus 53 in the opposite direction, and retains 95.67% of the similar model's solved questions.

• The gain survives harder problems and different students. The advantage extends to problems harder than those used in either training stage and also appears with Qwen3 students: at both Qwen3 model sizes and both selection budgets (50,000 and 200,000 SFT rows), Diverse leads on every reported metric, including extrapolation problems above the difficulty used in either SFT or RL.

• Multi-teacher sourcing is a coarse proxy for the same effect. On Enigmata, for Qwen3-4B-Base, twelve teachers add about 18 points of pass@64 over one teacher at either environment pool size, while enlarging the pool from 16 to 399 RLVE environments adds up to 6.4 points at a fixed teacher count and under half a point at twelve teachers. On MATH-500 in the 16-environment pool, pass@1 rises from 34.14% to 65.08%. Across seven sampling budgets from pass@1 to pass@64, twelve teachers lead in 54 of the 56 benchmark, pool, and budget cells. For Qwen3-1.7B trained with one to five teachers and then RL on DAPO-Math-17k, multi-teacher conditions beat one-teacher in all 32 comparisons at pass@1 and 27 of 32 at pass@64.

• The single-model condition reproduces the benefit. Qwen3-4B-Thinking-2507 writes every candidate, and route-diverse selection from that one pool improves mean pass@8 across 10 mathematics benchmarks by 3.39 to 6.17 points at three selection budgets (10k, 25k, and 50k examples; evaluated at RL steps 50, 30, and 30 respectively).

• A pre-RL diagnostic explains why. With binary rewards, a rollout group whose attempts all succeed or all fail yields zero group-relative advantage. On 64 mathematics prompts, the route-diverse OLMo3-7B checkpoint produces mixed outcomes on 54.7% of prompts, compared with 46.9% for the route-similar checkpoint and 51.6% for the pre-SFT base, despite slightly lower mean accuracy. The two selections move this share in opposite directions from the base.

• Post-RL completions are more varied in wording. After the same RL on RLVE, the correct completions of the diverse Qwen3 checkpoints have mean bigram Jaccard distance 16.68% higher for Qwen3-4B and 15.13% higher for Qwen3-1.7B than those of the similar checkpoints.

• The method wins on real corpora and costs far less. Applied to OpenThoughts3, INTELLECT-3, and Nemotron-Cascade 2, the selector beats random selection, a topology baseline, and gradient-diversity, embedding, and lexical selection in every comparison of mean post-RL accuracy and pass@8, with relative gains from 1.2% to 10.8% at pass@1 and pass@8 (evaluated at RL step 64). Against the similar selection from the same pool, it leads on every mathematics benchmark in all three corpora by 4.9 to 18.5 points of average accuracy, and also leads on all three OMEGA splits and GPQA-Diamond. On a pool of about 2.1 million solutions it takes about three hours on one CPU node and no GPU time, while the gradient-diversity and embedding baselines pass every candidate through a 7B or 8B model and need 64 to 232 GPU-hours.

• Coverage gaps are largest near the edge of what a model solves reliably. The gap peaks at intermediate difficulty for Qwen3-4B-Base, on the easiest band for OLMo3-7B, and at the smallest Qwen capacity on OMEGA — consistent with the account that the gap should be largest on problems near the boundary of reliable solving.

Methodology in Plain English

The authors start from a fixed pool of verified solutions, meaning candidate answers that a checker has confirmed are correct. Each solution is a route: an ordered sequence of reasoning steps leading from the problem to the answer.

To compare routes without reading them with a language model, they build a fingerprint. The reasoning text is split into steps at blank lines, discourse markers, and sentence boundaries. A fixed first-match rule assigns each step a type from a dataset-specific vocabulary — for example, the released corpora, Dolci-Think, and the single-model pool use ten types (setup, computation, deduction, verification, backtracking, exploration, backward reasoning, decomposition, commentary, conclusion), RLVE uses 18 note types, and OMEGA reads 15 annotated step types plus one of 32 strategy labels and 13 cue features.

Those type sequences become a directed transition graph and a reasoning tree, where exploration steps open branches, verification steps attach as leaves, and backtracking steps re-attach under earlier nodes. The fingerprint concatenates up to five blocks: continuous statistics (frequencies, transition rates, topology, motifs, tree shape), tree features (per-edge-type transition matrices, depth-tiered distributions, hashed root-to-leaf paths), binary strategy-pattern indicators, dense conversation-level statistics, and a signed feature-hashing bag of route labels using SHA-256. A fixed random projection shortens the vector. Distances in this space approximate differences in procedure and can also reflect wording.

For a target budget n, the diverse set clusters the fingerprints, assigns each cluster a size-proportional budget, and repeatedly picks the candidate farthest from those already chosen. The matched similar set is built by nearest-centroid selection from one dense region. Both sets are then given the same SFT recipe and the same RL recipe and evaluated at the same RL step.

The experimental design holds everything else constant. The two conditions share the student model, prompt pool, SFT trajectory budget, group-relative RL recipe, evaluation protocol, and checkpoint step; only the retained solutions differ. On RLVE, SFT covers difficulty 1 to 5, RL extends through difficulty 10, and evaluation runs to difficulty 15. The environment split marks 63 evaluation environments as Seen and 321 held out from SFT as Unseen, with the shared RL pool spanning all 384 environments. Evaluations report sampled coverage (pass@k), and the pre-RL reward-signal diagnostic is measured before RL begins.

Why This Matters

Impact on research. The paper reframes SFT data selection as a question about which procedures a model practices, not just which problems it sees answered correctly. It connects data curation to group-relative RL mechanics: because a rollout group with uniformly correct or uniformly wrong rewards contributes no advantage signal, a starting policy that puts a correct solution within sampling reach on more prompts gives RL more to learn from. The authors position this as acting before RL, complementing methods that filter zero-variance groups, maintain rollout diversity, or select prompts by reward variance. It also provides a cheap alternative to selection methods that call a language model or a gradient pass on every candidate.

Real-world applications:

• Building post-training pipelines where a verifier produces far more solutions than the SFT budget can consume, and a subset must be chosen.
• Curating released reasoning corpora such as OpenThoughts3, INTELLECT-3, and Nemotron-Cascade 2 before fine-tuning a student.
• Domains with programmatic verifiers and replayable or executable routes — synthetic puzzles, Sokoban, and program simulation are all evaluated in the paper, and the authors note the gap grows with the number of samples there.
• Compute-constrained settings, where selection must run without GPUs: the method processes a pool of about 2.1 million solutions in about three hours on one CPU node.

Industry relevance. The selector avoids model calls, additional generation, and gradients, which matters when candidate pools are large and GPU time is the bottleneck. The reported cost contrast — about three CPU hours versus 64 to 232 GPU-hours for gradient-diversity and embedding baselines — is directly relevant to teams deciding how much of their post-training budget to spend on data curation.

Future Directions

• Why route diversity helps, formally. The paper offers a pre-RL account based on mixed-reward probability and an illustrative model of how the coverage gap widens with sampling budget and then narrows at saturation. What is not established is a general predictive theory linking fingerprint spread to RL learnability.

• Extending beyond the studied settings. Results cover RLVE, Enigmata, OMEGA, reasoning-gym, DAPO-Math-17k, Sokoban, and program simulation, plus mathematics, code, instruction following, science, puzzle, and instruction-following benchmarks. How far the effect holds in other domains, model families, and RL algorithms is not reported.

• Trade-offs between diversity and accuracy. The diagnostics show the route-diverse checkpoint has mixed outcomes on more prompts despite slightly lower mean accuracy. The paper reports this pre-RL observation; how to balance the two signals during selection is left open.

• Selection at larger scale and in combination with other interventions. The authors frame route selection as complementary to reward-shaping methods inside group-relative RL, prompt selection by reward variance, and other data-side techniques. Whether these stack, and how, is not reported in the truncated content.

Target Audience

Researchers and engineers working on LLM post-training and reasoning — particularly those designing SFT-and-RL pipelines, curating reasoning datasets, or studying data selection. It is also relevant to readers interested in how a starting policy constrains what reinforcement learning can discover under a finite rollout budget. Readers should be comfortable with pass@k metrics, group-relative RL, and basic representation-distance concepts; no deep gradient-based analysis is required to follow the argument, since the proposed method itself is rule-based and text-only.

Authors’ abstract

Verified solutions are not equally useful for preparing reasoning models for reinforcement learning (RL). We present a comprehensive study of route diversity, the variation in the sequences of reasoning steps in supervised fine-tuning (SFT) data, and propose a lightweight, rule-based fingerprint to select for it. From one pool at one budget, with matched training recipes and checkpoints, selecting diverse rather than similar routes improves post-RL problem coverage across puzzles and mathematics, including on problems harder than those seen in either training stage. In synthetic experiments, route-diverse SFT improves OLMo3-7B's pass@8 by 16.9 points on environments held out from SFT. In a single-model condition, where one model writes every candidate, diverse selection gains up to 6.2 points of mean pass@8 across 10 mathematics benchmarks. Pre-RL diagnostics suggest why: diverse SFT can produce both successful and failed attempts on more prompts despite slightly lower mean accuracy, giving group-relative RL more prompts with a learning signal. On 3 open-source corpora, our CPU-only selector, without model calls, outperforms more expensive alternatives in every comparison of mean post-RL performance. These results identify reasoning-route diversity as a practical criterion for selecting SFT data that better prepares models for RL.