Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

AI summary

Overview

Research area: Natural Language Processing — specifically knowledge distillation and supervised fine-tuning of large language models for reasoning.

Technical level: Intermediate. Readers should be comfortable with transformer language models, supervised fine-tuning (SFT), and basic information-theoretic concepts like log-likelihood.

One-sentence scope: The paper introduces Rank-Surprisal Ratio (RSR), a simple, single-forward-pass metric that predicts how useful a given teacher model's reasoning trajectory will be for a specific student model, and validates it across 55 teacher-student pairings.

What This Paper Is About

When researchers train smaller "student" language models on long chain-of-thought reasoning traces generated by larger "teacher" models, they often assume that stronger teachers produce better students. This paper shows that assumption is wrong: the effectiveness of a teacher's data depends heavily on the specific student. The authors propose a simple metric, RSR, that predicts which reasoning trajectories will actually improve a given student by jointly measuring how informative and how well-aligned those trajectories are with the student's existing behavior.

Key Contributions

1. A large-scale distillation study covering 5 student models and 11 teacher models (55 pairings) that demonstrates reasoning trajectory effectiveness is highly student-specific, with no reliable rule based on teacher size or model family.

2. The Rank-Surprisal Ratio (RSR) metric, which quantifies data-student suitability as the ratio between a trajectory's average token rank and its average surprisal under the student model. It requires only a single forward pass, no verifier or held-out test data, and is straightforward to interpret.

3. Strong empirical validation: RSR achieves an average Spearman correlation of 0.86 with post-training reasoning performance across all five students, substantially outperforming probability-based, rank-based, gradient-based, and LLM-judged alternatives.

4. Two practical applications: RSR-based trajectory selection (picking the best of 33 candidate trajectories per problem) and teacher selection (identifying the best teacher from 200 sample trajectories), both beating existing methods and approaching oracle performance.

Main Findings

• Stronger teachers do not produce better students. The 671B DeepSeek-R1 and 235B Qwen-3-Thinking teachers frequently underperform the much smaller 32B QwQ on several students. Neither parameter count nor teacher reasoning accuracy reliably predicts student improvement.

• Data-student suitability is critical and student-specific. Model family distance matters too — teachers from distant families (e.g., GPT-OSS for Qwen students) often transfer poorly. No simple surface attribute explains which pairing works.

• Probability-based metrics favor the wrong data. Metrics like average surprisal and local surprisal assign the lowest surprisal to trajectories from Nemotron-Super and Magistral-Small, yet training on those trajectories yields poor reasoning performance. High-likelihood trajectories tend to be familiar but uninformative.

• Effective trajectories combine absolute unfamiliarity with relative familiarity. Suitable trajectories have low absolute probability under the student (high surprisal) while their tokens still rank relatively high in the model's vocabulary distribution. This resolves the apparent dilemma between informativeness and alignment.

• RSR correlates strongly with post-training performance. Average Spearman correlation of 0.86 across the five students, versus at most 0.59 for any competing metric (including G-Norm, GRACE, influence scores, LLM-judged quality, and rule-based quality).

• Both design components matter. Ablations show that removing rank clipping drops correlation from 0.856 to 0.700, and removing the surprisal-weighted average drops it to 0.391. RSR is robust to reduced sample size (200 trajectories per teacher gives 0.864 correlation).

• RSR selection beats single-teacher brute-force search. In trajectory selection, datasets built by RSR outperform all baselines and match or exceed the best single-teacher performance for four of five students — a strong upper bound obtained by exhaustive search.

• RSR works in low-resource teacher selection. Using only 200 trajectories per candidate teacher, RSR's top-1 and top-2 teacher picks approach oracle performance (average 48.3 vs. oracle 48.7), outperforming all other metrics.

Methodology in Plain English

The authors first built a controlled testbed. They collected 5,000 math problems from NuminaMath and had 11 different reasoning models (ranging from 4B to 671B parameters) generate long chain-of-thought solutions for each problem. Then they fine-tuned five different base student models on each teacher's dataset, producing 55 distinct student datasets, and measured each student's reasoning ability on four math benchmarks (AIME 2024, AIME 2025, AMC 2023, MATH500) using the Acc@4 metric, averaging over three independent generation runs.

Their key insight came from a simulation. They modeled the student's token-level prediction distribution as a mixture of two Zipf distributions — a dominant "major mode" representing the model's usual generation patterns, and a smaller "minor mode" representing less common but still familiar patterns. Four trajectory types were simulated: ones following the major mode, ones following the minor mode, ones from a completely unrelated distribution, and ones drawn from the overall mixture. The minor-mode trajectories turned out to be the golden type: high surprisal (low absolute probability) but low rank (still near the top of the model's predictions).

This motivated the metric. At the token level, RSR is simply rank divided by surprisal. Naively averaging this per token is unstable, because tokens with near-zero surprisal produce division blowups. So the authors instead use a surprisal-weighted average, which algebraically simplifies to the sum of token ranks divided by the sum of token surprisals across the whole trajectory. They also clip very large rank values at a threshold (r_max = 100) because extremely unfamiliar tokens are effectively indistinguishable to the student anyway. The resulting trajectory-level RSR is a single number: lower means better suited.

To validate, they computed RSR for every teacher dataset under every student and measured its Spearman correlation with post-training performance. They then tested RSR as a selection criterion in two scenarios: picking one trajectory out of 33 candidates per problem, and picking the best teacher from six candidates using only 200 sample trajectories.

Why This Matters

Research impact. The paper reframes data selection for distillation as an informative alignment problem rather than a likelihood-maximization problem. It shows that the two signals — informativeness (surprisal) and alignment (rank) — are complementary and must be combined, which explains why existing metrics plateau. The finding that teacher quality does not transfer predictably also challenges a common assumption in the distillation literature. Because RSR requires only a single forward pass and no auxiliary verifier, it is easier to deploy than gradient-based (G-Norm, GRACE) or influence-function alternatives.

Real-world applications:

• Distillation pipelines: selecting which teacher-generated data to fine-tune on, reducing wasted compute on unsuitable trajectories.
• Cost-efficient teacher evaluation: identifying a good teacher for a given student using just 200 sample trajectories instead of generating and testing full datasets.
• Low-resource reasoning model training: enabling effective fine-tuning when only limited data or compute is available.
• Data curation for mixed-quality pools: filtering trajectories when candidates come from many heterogeneous sources.

Industry relevance. Companies training smaller, cheaper reasoning models via distillation can use RSR as a lightweight pre-filter before costly SFT runs. The metric's low computational cost (one forward pass, no gradients, no verifier model) makes it attractive for large-scale data engineering where per-example overhead matters.

Future Directions

1. Guiding trajectory rewriting rather than selection. When no candidate trajectory is well-suited to the student, the authors suggest using RSR to drive synthesis or rewriting of new trajectories instead of picking from a fixed pool.

2. Theoretical grounding. RSR is derived empirically and by simulation; the authors have not yet found a formal theoretical framework characterizing why the rank-to-surprisal ratio is the right functional form.

3. Cross-domain generalization. Experiments focus on math reasoning, with only a supplementary GPQA evaluation. Applying RSR to code generation, commonsense reasoning, and general text remains untested.

4. Subset selection across heterogeneous problems. Trajectory selection compares candidates for the same problem; subset selection requires cross-problem RSR comparison, which introduces confounding factors the authors flag as an open challenge.

Target Audience

This paper is most valuable to researchers and engineers working on LLM distillation, chain-of-thought data curation, and supervised fine-tuning for reasoning. It will also interest practitioners building training data pipelines who need a cheap, reliable way to rank candidate training trajectories, and to anyone studying the relationship between teacher and student capabilities in knowledge distillation. Readers without background in language model training or information theory may need to consult introductory material on log-likelihood and token-level decoding first.

Authors’ abstract

Long chain-of-thought (CoT) trajectories provide rich supervision signals for distilling reasoning from teacher to student LLMs. However, both prior work and our experiments show that trajectories from stronger teachers do not necessarily yield better students, highlighting the importance of data-student suitability in distillation. Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model's current behavior but overlooking more informative ones. Addressing this, we propose Rank-Surprisal Ratio (RSR), a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. RSR is motivated by the observation that effective trajectories typically balance learning signal strength and behavioral alignment by combining low absolute probability with relatively high-ranked tokens under the student model. Concretely, RSR is defined as the ratio of a trajectory's average token-wise rank to its average negative log-likelihood, and is straightforward to compute and interpret. Across five student models and reasoning trajectories from 11 diverse teachers, RSR strongly correlates with post-training reasoning performance (average Spearman 0.86), consistently outperforming existing metrics. We further demonstrate its practical utility in both trajectory selection and teacher selection.