SkillFactory: Self-Distillation For Learning Cognitive Behaviors

AI summary

Overview

Research area: Natural language processing, specifically post-training methods for LLM reasoning (supervised fine-tuning, reinforcement learning, and self-distillation).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (GRPO), SFT data pipelines, chain-of-thought reasoning, and terms like knowledge distillation and catastrophic forgetting.

Scope: The paper introduces SkillFactory, a method that restructures a base model's own sampled outputs into "silver" SFT traces demonstrating retry and reflection behaviors, then uses RL to sharpen those behaviors, evaluated on Countdown, OpenThoughts, and a suite of math and reasoning benchmarks across three base models.

What This Paper Is About

Reasoning models benefit from cognitive skills such as verifying answers, backtracking, and retrying with an alternate method, and reinforcement learning can amplify these skills when a base model already shows them. The problem is that many base models do not exhibit these skills in the first place, and the usual fix—distilling traces from a stronger teacher model—requires access to a superior model and often generalizes poorly beyond the distilled domain. SkillFactory asks whether a model can instead be primed to use these skills by rearranging its own correct and incorrect solution attempts into structured training traces, prior to RL.

Key Contributions

1. Skill acquisition without a teacher model. The paper shows that across two training settings (Countdown and OpenThoughts), models can acquire complex reasoning skills from their own rearranged outputs rather than from distillation of a stronger model.
2. Better easy-to-hard generalization after RL. SkillFactory initialization enables generalization to harder task variants and novel domains post-RL, matching or exceeding strong baselines that use external data curation or distillation.
3. Improved robustness out-of-domain. SkillFactory models show greater resilience to catastrophic forgetting and regression on out-of-domain tasks than RLed base models.
4. Trace-construction algorithm and analysis. The paper specifies an algorithm for building traces from correct and incorrect solution-reflection pairs, plus an analysis quantifying the number of explicit answer attempts, explicit reflections, and verifier F1, alongside ablations of sample ordering, reflections, and prompt diversity.

Main Findings

• Lower SFT accuracy, higher post-RL accuracy. On Qwen2.5-1.5B-Instruct trained on Countdown-3arg, R1 Distillation achieved much higher SFT accuracy than SkillFactory (11.7% vs. 2.8%), but after RL the relationship reversed: SkillFactory→GRPO reached 25.1% on harder Countdown variants versus R1 Distill→GRPO at 21.2%, a +3.9 point gap, and improved over RL-only by 9.3%.

• Easy-to-hard Countdown generalization. SkillFactory→GRPO's 25.1% was the highest accuracy among all methods on held-out Countdown 4–6-argument variants, with RL-Only at 15.8%, STaR→GRPO at 9.7%, BOLT→GRPO at 13.7%, and the base Qwen2.5 1.5B Instruct at 1.9%.

• Comparable but not superior OOD average. On out-of-domain tasks (Acronym, Letter CD, Multiplication, CSQA, GSM8k), R1 Distill→GRPO slightly surpassed SkillFactory→GRPO overall (35.9% vs. 35.7%), while both exceeded RL-Only (31.9%) and BOLT→GRPO (33.0%).

• Competitive results on challenging math. Training Qwen2.5-7B-Instruct on OpenThoughts, SkillFactory with 10k rows reached 40.6% overall versus QwQ distillation with 10k rows at 42.5% and RL Only at 38.0%. At 1k rows, SkillFactory reached 42.1% overall and surpassed QwQ distillation on AMC (37.5%) and Math500 (64.6%). QwQ distillation showed degradation on Math500 relative to the base model even at 10k.

• Performance plateaus with more SFT data. SkillFactory's overall score decreased slightly from 1k to 10k examples (42.1% to 40.6%), which the authors attribute to the core skills being learned early, unlike distillation where new strategies and knowledge come from the teacher.

• Budget forcing helps SkillFactory on Countdown. With budget forcing, SkillFactory improved on Countdown from 17.5 to 22.8 (+5.3 points), outpacing RL-only (+1.2) and R1 distillation (+4.7). RL-only benefited most on Multiplication (+6.9, 24.6 to 31.4) versus SkillFactory's +1.3 (35.2 to 36.6), which the authors attribute to SkillFactory already performing retries and verifications during standard inference.

• Skills are actually used and reflection works. On Countdown-3arg, SkillFactory traces averaged 1.59 explicit answer attempts and 1.24 explicit reflections with verifier F1 of 0.96 for the correct class and 0.92 for the incorrect class. The incorrect-class F1 stayed above 0.8 across reported tasks, meaning wrong answers were correctly rejected. Countdown-4arg showed more reflection than Countdown-3arg (2.34 attempts, 7.13 reflections).

• Longer, more varied traces. SkillFactory induces much longer thinking traces than the RL baseline, making the output token-length distribution closer to that of an R1-distilled model, for both in-domain (Countdown-4arg) and out-of-domain (Multiplication, Letter Countdown) tasks.

• Ablations confirm the structured traces matter. On Qwen2.5-1.5B-Instruct trained on Countdown-3arg, OOD overall accuracy was 32.0% for SkillFactory, versus 25.8% for an instruction prompt variant, 24.1% for No Sample Order, 23.3% for No Reflections, and 29.0% for No Prompt Diversity.

• Ordering matters for verification. The No Sample Order ablation had higher verifier F1 in-domain on Countdown-3arg (0.99/0.97 versus 0.96/0.92) but suffered out-of-domain, e.g. Letter CD 4o at 0.22/0.65 and Mult 3dig at 0.22/0.74 versus SkillFactory's 0.34/0.82 and 0.35/0.81.

Methodology in Plain English

SkillFactory has three stages.

Data curation. For each training question, the base model is sampled with four different chain-of-thought prompts, 16 responses per prompt, yielding 64 solution attempts per question. Each attempt is automatically graded by extracting the answer from <answer> tags and comparing it against ground truth. The model is then prompted to reflect on each attempt and predict whether it is correct, using <verdict> tags; only reflections whose verdict matches the actual correctness are kept. Finally, correct and incorrect solution-reflection pairs are combined into a trace: a number of correct pairs and incorrect pairs are sampled, all but one correct pair is shuffled together with the incorrect pairs, and a correct pair is appended at the end so the trace always finishes successfully. Solutions and reflections are wrapped in <sample> and <reflect> tags and joined by transition phrases like "Let me reconsider."

Supervised fine-tuning. The model is fine-tuned on these silver traces. The authors explicitly state that improved task performance is not the goal at this stage; the aim is to internalize the patterns of sampling, reflecting, and retrying so RL can refine them.

Reinforcement learning. GRPO is applied with only binary correctness rewards, no KL divergence penalty, on held-out questions. The paper uses Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, and Olmo-3-7B-Instruct. The Countdown-3arg setting uses 4,000 rows for SFT data and 1,000 held-out questions for RL. The OpenThoughts setting uses 1,000 or 10,000 rows for SFT data and 10,000 held-out rows for RL.

Baselines are RL Only (GRPO from the base model), BOLT-style external data curation, R1 distillation, and STaR, each with an SFT and, where applicable, an RL stage. Evaluation covers Countdown 4–6 argument variants, Acronym (4, 5), Letter Countdown (4, 5), Multiplication (2, 3, 4, 5 digit), CommonsenseQA, GSM8K, GPQA, AIME 2025, AMC, and Math500. Most benchmarks are sampled 4 times; GPQA, AIME, and AMC are sampled 34 times because of their small size. A separate inference-time experiment appends a model-specific trigger phrase (for SkillFactory, a <sample> tag) after a 4,096-token generation to request another reasoning attempt, continuing up to 8,192 tokens total.

Why This Matters

• Research impact: The paper argues that inductive biases learned before RL may matter more than maximally learning the task during SFT, since higher pre-RL accuracy did not translate into better post-RL performance. It also offers a route to instilling reasoning skills that are unattested in a base model without access to a stronger teacher.

• Real-world applications:

  • Training reasoning models in settings where no superior teacher model is available or permissible, such as proprietary or regulated domains.
  • Improving reliability on search-focused tasks, where answers are easier to check than to generate, via explicit verification and retry behavior.
  • Inference-time scaling: budget forcing let SkillFactory gain +5.3 points on Countdown, and the authors note it can break a model out of degenerate repeated-output loops.
  • Preserving general capabilities: SkillFactory models showed greater resilience to regression on out-of-domain tasks than RLed base models.

• Industry relevance: The method reuses the model's own samples and standard GRPO, avoiding the cost of curating distillation data from a larger model. It also speaks to two practical failure modes named in the paper—overthinking (excessive verbosity) and underthinking (prematurely abandoning solution paths)—which directly affect inference cost and accuracy in deployed reasoning systems.

Future Directions

• Why more SFT data does not help. SkillFactory's score dipped from 42.1% at 1k rows to 40.6% at 10k rows; the authors suggest skills are learned early, but how to scale the recipe remains open.
• Domain-specific skill limitations. On Letter Countdown, low verifier F1 (0.34/0.82) was traced to the model's uncertainty about what is and is not an English word, suggesting a limitation of model scale; whether larger models or better lexical grounding resolve this is untested.
• Extending beyond retry and reflection. The paper frames SkillFactory as a platform for shaping cognitive behaviors broadly and argues that correcting skill use could be a path to avoiding overthinking and underthinking, but only retry and reflection are instantiated here.
• Ablation tradeoffs. The No Sample Order variant had higher in-domain verifier F1 but much worse out-of-domain F1, indicating the design of trace ordering is not fully understood.

Target Audience

Researchers and engineers working on LLM post-training, reasoning-model development, and RL fine-tuning who are interested in alternatives to distillation from stronger models. It is most useful to readers comfortable with GRPO, SFT pipelines, and benchmark evaluation, and to practitioners who need reasoning gains without access to a larger teacher model.

Authors’ abstract

Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL;(2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.