TraceForge: Distilling with Your Own Thinking

AI summary

Overview

Research area: Large language model distillation, synthetic reasoning-data generation, and post-training (supervised fine-tuning plus adversarial/preference optimization).

Technical level: Intermediate. The paper assumes familiarity with distillation, supervised fine-tuning, preference optimization (DPO), and best-of-N selection, but its argument is stated plainly enough for a strong newcomer to follow.

Scope: A single-author study (Yan Wang, Independent Researcher; arXiv:2610.29680v1, 21 Oct 2026, CC BY 4.0) proposing a two-stage generator-plus-critic pipeline that synthesizes "family-native" reasoning traces for a student model, and a five-corpus, five-benchmark controlled comparison of students trained on Qwen3.5-9B-Base.

What This Paper Is About

Frontier model providers now return only an answer or a short summary, withholding the reasoning trace that produced it, so the standard "distil the best teacher" recipe is blocked. A recent workaround replays providers' encrypted reasoning blocks across models to recover hidden traces verbatim (Panfilov et al. [2026]), but that violates provider terms of service and cannot support a product.

TraceForge proposes the legal alternative: don't recover the teacher's reasoning, generate it. Given the teacher's context and answer, a trained generator writes the derivation in the style of the student's own model family, and a critic keeps those traces indistinguishable from real family traces. The paper's question is whether a student trained on these synthesized traces is as good as, or better than, a student trained on the teacher's genuine reasoning.

Key Contributions

1. The "family-native" trace. A formulation of the missing-data problem as learning P(R | S, U, A, T) — the reasoning trace R a model would write given the system prompt S, user turns U, answer A, and any tool calls or responses T — with the target trace always drawn from the student's own model family, never from the closed teacher.

2. A two-stage generator/critic pipeline. Supervised fine-tuning of a generator on roughly ten thousand Qwen3.8-Max conversations (trace removed from input, loss on the trace only), followed by supervised training of a critic on matched real/generated pairs, followed by five rounds of online adversarial co-training in which fooling traces become critic data and failing traces become DPO preference pairs for the generator.

3. A controlled five-corpus comparison. Five training corpora built from one fixed pool of 5,000 Claude Code trajectories (256K context) that share prompts and Fable 5 answers, varying only the source of the reasoning trace and the answer, with the base model, recipe, and budget held fixed across five benchmarks (ARC-Easy, ARC-Challenge, BoolQ, HumanEval+, MBPP+).

4. An empirical reversal of the standard assumption. On the two coding benchmarks, synthesized family-native traces outperform the strongest teacher's own traces and substantially outperform the strongest legally distillable model in the student's family — plus a fully "own-data" variant in which even the answer is written by Qwen3.8-Max so that no Claude or OpenAI token is ever backpropagated.

Main Findings

• Family-native traces beat the teacher's real traces. With the same questions and the same Fable 5 answers, replacing Fable 5's reasoning trace with a family-native one raises HumanEval+ by 3.7 points and MBPP+ by 7.1 points at the final checkpoint (S1 → S2: 64.63 → 68.29 and 52.38 → 59.52). The MBPP+ lead holds at every checkpoint measured.

• The traces pass a closed judge. GPT-6.1-SOL scores real family reasoning traces at 0.66 and the generator's traces at 0.68 after adversarial training, rising to 0.82 when the critic selects the best of four candidates. After supervised training alone the generator averaged 0.54, against 0.66 for real traces.

• Teacher capability is necessary but not sufficient. Training on Qwen3.8-Max's own reasoning traces and answers (S3) trails the family-native corpus by 21 points on HumanEval+ and 35 points on MBPP+ (Section 4 and Table 1; Section 3.2's prose states this comparison as 18 and 28 points). S3's final scores are 46.95 and 24.60.

• The synthesizer matters more than the teacher's answer source. With identical GPT-6.1-SOL answers, family-native traces (S5: 59.15 and 60.05) beat TraceInversion (S4: 3.66 and 6.61) by 55 and 53 points with nothing varying but the synthesizer. S5 sits 9 points behind S2 on HumanEval+ and level on MBPP+, roughly the expected gap between the two teachers.

• Non-reasoning tasks do not separate the corpora. On ARC-Easy, ARC-Challenge, and BoolQ every value sits within 1.6 points of every other, and no setting leads on all three; the code benchmarks, by contrast, spread by more than sixty points.

• Foreign-style traces destabilize training. S1 reaches 66.46 on HumanEval+ at step 600, falls to 33.54 at step 1,200, and recovers to 64.63 at step 1,600; the 1,200-step evaluation was re-run twice and held. The author reads this as a long parameter move from fitting a foreign style that can overshoot.

• Distilled-from-distilled corpora collapse into loops. S3 peaks at 48.78 on HumanEval+ within the first 300 steps and never recovers; S4 peaks at 21.95 and 42.59 at step 900 before falling to 3.66 and 6.61. The author attributes this to dead loops and repetitive re-derivation, the pattern colloquially called "breathing second-hand smoke." S1 never shows it, and the family-native traces do not produce it at all.

• Family-native traces improve steadily. S2 climbs monotonically to 68.29 and 59.52; S5 rises at every single checkpoint on both benchmarks, from 0.61 and 10.32 to 59.15 and 60.05, and edges past S1 on the average of the two. S5's slow start is attributed to GPT-6.1-SOL's answer style being unfamiliar to Qwen3.5's mid-training.

• A fully self-contained variant works. Generating the family-native trace, handing it to Qwen3.8-Max to write the answer, and training on both performs "essentially the same as S2" — no Claude or OpenAI token is backpropagated at any point.

• Reported caveats. One run per cell, so no variance or confidence intervals are reported. The abstract states "the same 3,500 questions," while Section 3.1 describes 5,000 Claude Code trajectories and 5,000 prompts.

Methodology in Plain English

The setup treats a conversation as a message list: system prompt, user turns, tool calls and responses, an answer, and the reasoning trace that precedes the answer. The goal is a model that can write the missing trace given everything else.

First, the generator — a model from the Qwen family — is fine-tuned on roughly ten thousand real Qwen3.8-Max conversations with the reasoning trace stripped from the input and the loss applied to the trace alone. Two design choices matter: the target is always a trace the family actually wrote (so style is learned from data rather than requested in a prompt), and the answer is always an input and never an output (so the model learns a derivation that ends at the answer instead of restating it). Because the training traces belong to the family's own answers, asking the generator for a trace to accompany a teacher's answer is not reproduction — it is the model doing exactly what it was trained to do.

Second, a critic is trained to separate real family traces from generated ones, using one matched negative per training sample, and training stops once the critic's margin reaches 0.2 rather than being pushed as far as possible. The author reports a failure here: a critic smaller than the generator fits shortcuts within a few hundred steps and gives the generator nothing to learn from, so the critic must be at least as large as the generator.

Third, five rounds of online co-training alternate between generator and critic, with 5,000 generated traces per round; traces that fool the critic become critic training data, traces that fail become preference pairs for the generator under standard DPO. A held-out set is re-scored with GPT-6.1-SOL each round, the score stops improving after five rounds, and the author stops there. At inference, the critic picks the best of several candidates with no labels.

The evaluation holds the base model (Qwen/Qwen3.5-9B-Base, full-parameter BF16 fine-tuning, AdamW with weight decay 0.1, gradient clipping 1, learning rate 2×10⁻⁵ under a cosine schedule with 5% warmup down to 2×10⁻⁶, micro-batch 1, eight H20 GPUs with TP4/PP2 and sequence parallelism, seed 42, no shuffling, no truncation, roughly 1,600 packed-batch updates) and the training recipe fixed, and varies only where the trace and answer come from across the five corpora S1–S5. Three multiple-choice benchmarks test non-reasoning general ability; two coding benchmarks test reasoning-mode coding.

Why This Matters

Impact on research. The paper reframes the distillation debate around a single testable question — not "how do we get around the restriction?" but "does the trace's form matter as much as its content?" Its answer, that a trace written in the student's own family voice teaches more than a stronger teacher's genuine trace with the answers held fixed, challenges the default assumption that the strongest available reasoning signal is the best training signal. It also offers a legal, reproducible alternative to encrypted-block replay, and documents a training-instability pattern (dead loops when distilling from an already-distilled model) that practitioners will recognize.

Real-world applications.

• Building coding assistants from a strong closed teacher's answers when its reasoning traces are unavailable or license-restricted.
• Producing compliant training data pipelines for organizations that cannot legally backpropagate third-party tokens, using the fully self-contained variant.
• Generating higher-quality synthetic reasoning data at lower cost than collecting premium traces, since the generator and critic are trained once on in-family data.
• Automated quality filtering: the critic's authenticity score doubles as a data-quality signal that can flag synthetic or degenerate traces before they enter a training set.

Industry relevance. Every team that distils from a frontier API faces the same terms-of-service and provenance constraints. This pipeline keeps the teacher's content while manufacturing the form in-house, which matters for legal review, for audit trails, and for anyone whose data-governance policy forbids certain tokens in the gradient. The reported instability of foreign-style and second-hand corpora is also directly actionable for teams running long fine-tuning jobs.

Future Directions

• Extending beyond one model family. The accompanying API currently serves Qwen-family traces only, with other families to follow — the central claim is untested on non-Qwen students, and the author's explanation rests on Qwen-specific mid-training data.
• Isolating "family style" from everything else. The paper itself notes that the S1/S2 and S4/S5 comparisons establish downstream differences but "do not isolate family style from every other difference between the pipelines." A controlled ablation of style alone is the obvious next experiment.
• Understanding and eliminating the dead-loop collapse. S3 and S4 both degenerate into repetition; the author offers an intuition rather than a diagnosis. A mechanistic account, and a method that makes any corpus loop-resistant, would be broadly useful.
• Statistical robustness. With one run per cell and no error bars, and with a single 9B base model and a single seed, replication across seeds, scales, and base checkpoints is needed before the effect size can be trusted.

Target Audience

Practitioners building distillation and synthetic-data pipelines; post-training and data-quality engineers at model labs and applied AI teams; researchers working on reasoning-trace supervision, preference optimization, and teacher–student distribution mismatch; and policy or legal reviewers who need to assess whether a training corpus was obtained within provider terms. Readers wanting the mathematical formulation or a benchmark-leaderboard result will find neither — the value here is the pipeline design and the controlled data-source comparison.

Authors’ abstract

Using an LLM to generate training data for another is among the most general ways to train one, and it remains common practice. Some frontier teams have closed that route for their own models: what comes back is an answer, or a short summary, without the detailed reasoning trace behind it. That is a little ironic: they were themselves built on distilling human knowledge, and distilling an open-source model is still entirely routine; the restriction applies to their own outputs alone. Recent work recovers the hidden reasoning by replaying encrypted blocks across models. It somehow works, but it breaks the providers' terms of service, so it is not something a product can be built on long-term. We built the alternative. We take the context and the teacher's answer as input, and generate the reasoning trace ourselves, in the style of the student's own model family. Training is two-stage, supervised and then adversarial: a generator learns to regenerate the family's own reasoning traces from their context and answer, and a critic learns how real a reasoning trace looks. Applied to a teacher's answer, the generator writes one in the family's style. An independent judge rates how likely traces are to be genuine on a scale from 0 to 1: ours average 0.68, real traces 0.66, and traces selected by the critic from four candidates 0.82. These are authenticity scores, not correctness rates. We test the idea on five benchmarks with everything else held fixed. On general tasks the five training corpora are about the same; on coding, the generated reasoning beats Fable 5's own thinking content. With the same 5,000 questions and the same teacher answers, HumanEval+ improves by 3.7 points and MBPP+ by 7.1 at the final checkpoint, and the MBPP+ lead holds at every checkpoint. We suspect that reasoning written in the student's family style is easier for it to learn from.