From Rollouts to Recipes: Self-Contained Post-Training for LLMs

AI summary

Overview

Research area: Post-training of large language models (LLMs) for reasoning, specifically verifier-based reinforcement learning on mathematical reasoning tasks.

Technical level: Advanced.

Scope: The paper proposes Self-Routing, a framework that uses a model's own on-policy rollout behavior (answer correctness and predictive confidence) to route each training sample to one of four training recipes — GRPO, on-policy self-distillation (OPSD), regularization (REG), or skipping (SKIP).

What This Paper Is About

Nearly all post-training methods for LLMs apply a single global recipe, or a fixed mixture of objectives, to every sample in the training set. The authors argue that the model's own rollouts already reveal sample-level learning states, and that a sample with mixed correct/incorrect rollouts needs a different optimization signal than one that always fails or is already solved reliably. Their goal is a self-contained training loop that converts these free-to-obtain rollout signals into per-sample training decisions, without an external teacher model, extra annotations, or additional sampling.

Key Contributions

1. Identifies a structural limitation of verifier-based post-training: global recipes cannot adapt to the heterogeneous sample-level learning states that current-policy rollouts expose.
2. Proposes self-contained behavior-conditioned routing: rollout correctness and confidence are turned into sample-level recipes at no extra annotation, evaluator, or sampling cost.
3. Shows empirically that different rollout behavior states benefit from different optimization mechanisms, and that routing improves over uniform GRPO, uniform OPSD, fixed objective mixtures, and filtering- or curriculum-based baselines.
4. Provides routing-dynamics and efficiency analyses: the recipe distribution shifts over training (OPSD early, GRPO mid, REG late), and expensive recipes are applied only to the samples that need them.

Main Findings

• Self-Routing achieves the highest average score on every evaluated backbone. The paper tests Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3.5-0.8B, Qwen3.5-2B, and Qwen3.5-4B, comparing against Base, Naive-GRPO, and Naive-OPSD. Averages for Qwen3-4B are Base 61.0, Naive-GRPO 66.8, Naive-OPSD 70.4, Self-Routing 73.7.

• The largest gains are on in-domain mathematics. On Qwen3-4B, Self-Routing improves the average from 61.0 to 73.7, exceeding Naive-GRPO and Naive-OPSD by 6.9 and 3.3 points respectively. On Qwen3.5-4B it reaches 86.6 average, versus 79.8 for Naive-GRPO and 83.0 for Naive-OPSD.

• Gains grow with model capacity. On the smallest models (Qwen3-0.6B, Qwen3.5-0.8B), Naive-GRPO and Naive-OPSD still show mixed results on individual benchmarks such as AIME24 and GPQA-diamond. For stronger backbones, OPSD becomes consistently better than GRPO and Self-Routing widens the margin further.

• Out-of-domain behavior differs by benchmark. GPQA-diamond usually improves after math-oriented post-training, while MMLU-Pro drops relative to the base model for all post-training methods. Self-Routing shows the smallest degradation and remains the second-best method after Base on every backbone. The authors conjecture this comes from the conditional nature of routing, which avoids applying a uniform math-oriented update to every instance.

• Self-Routing also beats stronger baselines on Qwen3-4B. It outperforms DAPO-style RL and PODS, achieving 80.9 / 71.1 / 59.3 on ID math, OOD verifiable reasoning, and general evaluation, and obtains the best average score of 71.1 on SATBench, AutoLogi, and LiveCodeBench-v5.

• Accuracy-based routing is strong, but confidence adds value. In the routing-strategy ablation on Qwen3-4B, round-wise random and fixed-ratio random routing (Reg:GRPO:OPSD:Skip = 3:3:3:1) perform poorly. Accuracy-Based Routing (lowest 30% to Reg, next 30% to GRPO, next 30% to OPSD, highest 10% to Skip) even outperforms Naive-OPSD, but still falls slightly behind Self-Routing across most benchmarks, indicating that accuracy alone is insufficient for recipe assignment.

• The method is not a wall-clock acceleration technique. At G = 8 rollouts, normalized FLOPs costs are 64.0 for Naive-GRPO, 24.0 for Naive-OPSD, and 34.7 for Self-Routing. Self-Routing is more expensive than Naive-OPSD but cheaper than applying GRPO to all rollout groups; the benefit is selective allocation rather than raw speed.

• Routing distribution shifts substantially during training. On Qwen3-4B, with routing decisions sampled every ten training steps, OPSD dominates early with a counting ratio above 0.5. GRPO rises and peaks in the middle stage, and REG becomes the dominant branch late in training while OPSD and GRPO decline. SKIP stays relatively low throughout and decreases near the end.

• Aggregate branch ratios. Integrating the routing curves gives 30.8% GRPO, 30.4% OPSD, 25.5% REG, and 13.3% SKIP.

Methodology in Plain English

The method keeps the standard post-training pipeline — sample prompts, generate multiple on-policy rollouts, score them with a binary verifier — and inserts a lightweight router between rollout collection and loss construction.

For each sample, two signals are computed. The first is rollout accuracy: the fraction of its G rollouts that the verifier marks correct. Rather than hard-thresholding this into low/medium/high buckets (which would be unstable when G is small), the authors map it to three smooth Gaussian membership scores centered at 0, 0.5, and 1, with fixed widths of sigma_l = sigma_h = 0.18 and sigma_m = 0.16, and normalize them.

The second signal is confidence, estimated from token-level predictive entropy of the generated response, averaged over the sequence, converted to confidence via batch-level normalization, then averaged across rollouts and calibrated against the batch mean.

These two signals feed four interpretable routing scores (one per recipe) that are normalized into a probability distribution, and each sample is drawn into exactly one queue: GRPO, OPSD, REG, or SKIP. GRPO uses the standard group-relative advantage with a clipped ratio. OPSD has the same model act as its own teacher, conditioned on the answer; the teacher target is generated once during offline preprocessing by prompting the base model with the problem and its ground-truth answer, then reused. REG applies a KL penalty keeping the policy close to a reference policy for samples that already behave reliably. SKIP contributes no gradient. The final loss is a size-weighted average over the three active queues, with skipped samples excluded from the denominator. The routing assignments were motivated by preliminary diagnostics on Qwen3-4B.

Training infrastructure is ms-swift, run on a single node with 8 NVIDIA A100 GPUs using PyTorch and the Hugging Face model/tokenizer interface. All compared methods share the same data loader, rollout interface, verifier, tokenizer, and checkpointing code. The training set is DAPO-Math-17K; evaluation uses the in-domain benchmarks GSM8K, MATH-500, AIME24, and AIME25, plus the out-of-domain benchmarks MMLU-Pro and GPQA-diamond, with a macro-average over all six.

Why This Matters

Impact on research. The paper reframes post-training as a routing problem rather than a single-objective design problem. Instead of asking which RL objective is best, it asks what optimization signal a given sample should receive given the model's current behavior. This complements existing work on data selection and curriculum learning, which mainly decides which samples to train or in what order, while this work asks how a sampled prompt should be optimized. It also positions on-policy self-distillation as a self-contained primitive for dense guidance under sparse rewards.

Real-world applications (plausible implications of the method, not claims measured in the paper):

• Mathematical reasoning assistants and tutoring systems that must improve on grade-school through competition-level problems.
• Code generation and program repair tools, where unit tests or execution feedback can serve the same verifier role as answer correctness.
• Formal-logic and verification workflows, given the reported transfer to SATBench and AutoLogi.
• General-purpose assistants that need to gain domain reasoning without losing broad benchmark performance, since the paper reports the smallest MMLU-Pro degradation among the compared post-training methods.

Industry relevance. The method requires no external teacher model, no reward model, no process annotations, and no additional sampling, and the routing module is a thin layer that fits into an existing training stack. That makes it practical for teams already running RLVR pipelines who want per-sample adaptivity without new infrastructure. The trade-off is that it is not faster than uniform OPSD under the current implementation, so the value proposition is training-signal quality rather than compute savings.

Future Directions

• Mechanistic or theoretical grounding. The authors state that the work does not provide a deep mechanistic or theoretical explanation for why certain rollout patterns align better with specific optimization signals; routing behavior is currently supported by intuition and experimental evidence rather than a rigorous understanding of the underlying optimization dynamics.
• Generalization beyond verifiable mathematical reasoning. Training experiments center on math; although transfer is observed to SATBench, AutoLogi, and LiveCodeBench-v5, whether the paradigm generalizes to agent planning or open-ended instruction following remains unclear.
• Improving the individual recipes. The paper deliberately does not improve the internal designs of GRPO-style RL (for example, reward engineering) or OPSD-like approaches (for example, trajectory refinement); these are described as largely orthogonal to the routing objective and remain open.
• Scaling behavior. Since the advantage over uniform baselines is reported to grow with model capacity across the tested backbones, the trend invites further study at scales beyond those evaluated.

Target Audience

This paper is most useful for machine learning researchers and engineers working on LLM post-training, reinforcement learning with verifiable rewards, and on-policy distillation. It will also interest practitioners who already run GRPO or OPSD pipelines and want sample-adaptive training without adding an external teacher or reward model, and researchers studying data selection, curriculum learning, or rollout-based uncertainty signals. Readers should be comfortable with reinforcement learning objectives, KL regularization, and token-level entropy, which is why the paper is rated Advanced.

Authors’ abstract

Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.