Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation

AI summary

Overview

Research area: Machine learning — knowledge distillation and reasoning in large language models (LLMs), with a focus on multi-teacher collaboration and step-level quality supervision.

Technical level: Intermediate. The paper assumes familiarity with knowledge distillation, Chain-of-Thought prompting, preference optimization (DPO), and reinforcement-learning-style training objectives, but the high-level ideas are explained in accessible terms.

Scope: A single framework paper proposing Collaborative Reasoning Distillation (CRD) — a three-component pipeline for building compact, high-quality reasoning training data from multiple teachers — together with benchmark results, ablations, and a stated limitations section.

What This Paper Is About

Distilling strong reasoning from large LLMs into small ones is hard: standard methods reward students based on whether the final answer is correct, so a lucky guess and a sound derivation get identical signals, and every reasoning flaw in a single teacher is inherited by the student. The authors build CRD, a framework in which two teachers critique each other's reasoning, a dual-judge system scores each individual reasoning step for logical validity, and a path-finding algorithm stitches the best steps from different teachers into coherent hybrid solutions that are then used to train compact student models.

Key Contributions

1. Interactive Multi-Teacher Cross-Feedback — an iterative loop in which N = 2 teachers (DeepSeek-R1 and Gemini 2.5 Flash Thinking) review, critique, and refine each other's reasoning chains, producing corrected and complementary candidate solutions rather than relying on simple ensemble voting.
2. Fine-grained Step-wise Quality Assessment — a dual-judge system (JudgeLM and xVerify, a 0.5B model) that assigns continuous quality scores to every reasoning step independently of whether the final answer is correct, using a 25% trimmed mean over four independent scores (each judge evaluates every step twice).
3. Coherence-Aware Step Stitching ("Thinking Path") — a Viterbi-style dynamic-programming search over a directed acyclic graph of reasoning steps that maximizes a combined objective of step quality and transition coherence (weighted by λ = 0.7), yielding hybrid chains no single teacher produced.
4. Reasoning Quality Optimization (RQO) with an Optimal Budget curriculum — a training objective that exponentially weights preference pairs by their quality margin, combined with a length curriculum that constrains token budgets progressively (32K → 10K → 1K).

Main Findings

• Headline benchmark results: CRD-4B reaches 97.3% on MATH-500, 76.2% on AIME'24, 70.3% on AIME'25, 57.4% on LiveCodeBench, and 58.7% on GPQA-Diamond, using only 50K training examples — described as up to 12 times smaller than datasets of comparable models.
• Small-model gains over the shared baseline: On DeepSeek-R1-Distill-Qwen-1.5B, CRD-1.5B improves MATH-500 from 83.9% to 91.1% (+7.2), GPQA-Diamond from 32.9% to 39.7% (+6.8), and LiveCodeBench from 16.9% to 27.2% (+10.3).
• Qwen3 student gains: CRD-1.7B improves over Qwen3-1.7B by +2.0 points on AIME'24 (48.3 → 50.3) and +5.4 points on AIME'25 (36.8 → 42.2).
• Large-scale student gains and teacher surpassing: CRD-30B-A3B improves AIME'25 from 70.9% to 73.2% (+2.3), LiveCodeBench from 62.6% to 65.9% (+3.3), and GPQA-Diamond from 65.8% to 67.2% (+1.4) over its Qwen3-30B-A3B base, and surpasses its own teacher DeepSeek-R1 on math and code benchmarks despite using 12 times fewer active parameters (3B vs 37B).
• Parameter efficiency: CRD-1.5B approaches the 7B distilled model on MATH-500 (91.1 vs 92.8, 4.7 times smaller), while CRD-1.7B (94.5%) surpasses the 14B distilled model (93.9%) on MATH-500 (8.2 times smaller). CRD-4B exceeds Hermes 4 70B (17 times smaller) and EXAONE Deep 32B on AIME'25.
• Data efficiency: CRD-1.5B reaches 35.1% on AIME'25 at 50K examples, versus DeepSeek-R1-Distill at 22.0% with 600K reasoning samples (12 times more data) and ProRL-1.5B at 33.3% with 136K samples (2.7 times more data). Within the curated dataset, CRD-1.5B improves monotonically from 28.1% at 10K to 35.1% at 50K.
• Curated data does most of the work: Naive SFT on the same 50K Thinking Path trajectories already reaches 90.5% on MATH-500; RQO adds a further consistent gain across all four benchmarks (+1.15 points on average), from 90.5 to 91.1 on MATH-500 and 33.4 to 35.1 on AIME'25.
• Cross-feedback improves raw teacher data: Comparing initial and refined outputs, mathematical reasoning accuracy rose from 64.1% to 83.5% and code generation success from 52.3% to 73.9%.
• Stitching rescues failed trajectories: Of 2,694 tasks with incorrect pre-stitch trajectories, 528 (19.6%) become correct after stitching.
• Ablation breakdown (DeepSeek-R1-Distill-Qwen-1.5B): Cross-Feedback contributes the largest single gain (+3.4 to +7.1 points across benchmarks), Thinking Path adds +1.1 to +3.5, Optimal Budget adds +0.2 to +0.6, and Judging adds +1.4 to +1.9, for total gains of +7.2 (MATH-500), +13.1 (AIME'25), +10.3 (LiveCodeBench), and +6.8 (GPQA-Diamond).
• Outperforms prior multi-teacher methods: Against TinyLLM, FAIR, and MAGDi on the same student backbones (Qwen-2.5-1.5B and Llama-3.1-8B) and the same teacher candidate pool, CRD leads by +18.5 points over MAGDi on MATH-500, +9.3 on GSM8K, and +8.8 on LiveCodeBench.
• Multi-teacher beats single-teacher: Single-teacher variants (Gemini-Flash-Distill, DeepSeek-R1-Distill) using the same scoring and training pipeline show particularly large CRD gains on AIME'24 (+14–18%) and LiveCodeBench (+10%).
• Lambda sensitivity: At λ ≤ 0.5, over-weighting coherence drops MATH-500 to 76.0%; at λ > 0.8, over-weighting local quality produces disconnected chains and 82.3%; performance peaks at λ = 0.7.
• Generalization beyond the training distribution: On BoxNet planning, CRD-1.5B scores 6.8 / 14.6 / 25.3 at pass@1 / pass@2 / pass@4 versus ProRL-1.5B at 7.1 / 13.8 / 22.4 and DeepSeek-R1-Distill-1.5B at 0.0 / 0.0 / 0.0. On the text-only HLE subset, CRD-30B-A3B improves over Qwen3-30B-A3B by 3.23 points (11.31 → 14.54).
• Training cost: The 1.5B student requires 1,632 training steps and 8.8 H100 GPU-hours, compared with 7,800 steps and 42 H100 GPU-hours for the 600K distillation baseline and more than 2K steps for ProRL.
• Step-level quality improvement: Algebra step correctness improves from 85.8% to 92.2%; cross-feedback converges after two rounds.
• Contamination checks: 11 DeepScaleR problems overlapped with MATH-500 (2.2%) and were excluded during 50% subsampling; AIME'24 and AIME'25 showed zero overlap; no code contamination was found across benchmarks, and LiveCodeBench v5 problems (August 2024–February 2025) post-date all code training sources (APPS 2021, CodeContests 2022, TACO 2023).

Methodology in Plain English

The pipeline has four stages.

1. Generate and refine. Two teachers (DeepSeek-R1 and Gemini 2.5 Flash Thinking) each produce an initial reasoning chain for a problem. Then, for K rounds, each teacher is shown the other's chain plus the original problem and asked, via a structured review prompt covering general criteria (logical flow, error detection, quality) and domain-specific criteria (algebraic validity, code compilability, quantifier scope, etc.), to find errors and produce an improved chain. The paper reports convergence at K = 2 to 3 rounds, and uses K = 2.
2. Score each step. Two judge models — JudgeLM and xVerify — each score every reasoning step twice, producing four scores. The minimum and maximum are discarded and the middle two averaged (a 25% trimmed mean), giving a continuous quality score per step. Domain-specific verification follows: code is executed and steps after failures are penalized, math steps after divergence from ground truth are penalized, and logic inference rules are checked.
3. Stitch the best chain. Steps are aligned across teachers using embedding similarity, with lookahead merging when one teacher's consecutive steps correspond to another's single step. The resulting graph is searched with Viterbi-style dynamic programming to find the path maximizing a weighted sum of step quality and transition coherence. This can produce a solution assembled from different teachers' strengths.
4. Train the student. The curated chains are used with Reasoning Quality Optimization, which weights preference pairs by an exponential function of their quality-score margin, so larger quality differences produce stronger learning signals. A curriculum progressively tightens the response-length budget from 32K to 10K to 1K tokens, with an exponential penalty for exceeding the current budget, so the student first learns correct long reasoning and then compresses it.

Training data totals 50K instances drawn from DeepScaleR (math), Eurus-2-RL (code), Reasoning Gym (logic), and SCP-116K (STEM). The paper notes that reasoning distillation shows a performance valley between 1K and 30K examples, and that 50K avoids it while remaining smaller than the 220K–800K used by prior methods.

Why This Matters

Impact on research. The paper argues that supervision quality — specifically step-level logical validity and cross-teacher synthesis — can partly substitute for parameter scaling and dataset scale. It provides a concrete alternative to outcome-based reward distillation and to chain-level multi-teacher ensembling, and its ablation isolates where the gains come from. It also demonstrates a student surpassing its own teacher on some benchmarks through cross-teacher synthesis.

Real-world applications.

• Latency-sensitive deployments where long reasoning chains are impractical but strong reasoning is still required.
• Code generation assistants that need step-by-step correctness, where the paper reports LiveCodeBench gains and execution-based verification.
• STEM tutoring or problem-solving tools built on compact models that can run at low cost.
• Edge or resource-constrained settings, since the method targets 1.5B–4B models with reduced training compute (8.8 H100 GPU-hours for the 1.5B student).

Industry relevance. The paper reports reduced data requirements (50K examples versus 220K–800K in prior work) and reduced student training cost, which matters for teams that cannot afford thousands of GPUs — the introduction cites estimated hardware costs in the $800 million range for cutting-edge models. It also shows gains generalizing across dense and mixture-of-experts architectures (Qwen3-30B-A3B).

Future Directions

• Replacing static, hand-crafted judging rubrics with learned quality models trained on preference data that can adapt to domain-specific nuances and scale better across domains.
• Extending the framework to heterogeneous teacher feedback to mitigate potential sycophancy in collaborative reasoning.
• Testing larger and all-open-weight teacher pools such as QwQ-32B and GPT-OSS, and stronger teachers for expert-level benchmarks such as HLE.
• Conducting controlled comparisons with on-policy distillation and SFT-based baselines, establishing formal bounds on stitching failures, running a larger human audit of stitched chains, and comparing RQO against DPO.

Target Audience

Researchers and engineers working on knowledge distillation, reasoning-model training, or efficient LLM deployment will get the most from this paper. It is also relevant to practitioners building small, cost-sensitive reasoning systems for math, code, or STEM, and to readers interested in LLM-as-a-judge evaluation and multi-agent or multi-teacher training pipelines. Readers unfamiliar with distillation or preference optimization would benefit from background reading first.

Note on scope: The paper reports one teacher pair (DeepSeek-R1 and Gemini

Authors’ abstract

Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggle to effectively distill reasoning to smaller models. Standard distillation methods rely on outcome-based rewards, failing to distinguish between sound reasoning and lucky guesses. We propose Collaborative Reasoning Distillation (CRD), a framework that enhances reasoning in compact models through three innovations: (1) interactive cross-feedback where teachers iteratively critique each other's reasoning, (2) fine-grained step-wise quality assessment capturing logical validity independent of final answers, and (3) coherence-aware step stitching that synthesizes complementary strengths. Students are trained via Reasoning Quality Optimization (RQO) with budget constraints. Our model, CRD-4B, achieves 97.3% on MATH-500 and 70.3% on AIME'25, surpassing baselines while using only 50K training examples, up to 12 times smaller than the datasets of comparable models.