Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

AI summary

Overview

Research area: Large language model training and reasoning, specifically post-training methods that sit between imitation learning (supervised fine-tuning) and reinforcement learning (RLVR). The paper also extends the method to agentic software engineering.

Technical level: Intermediate. Readers need some familiarity with supervised fine-tuning, reward-based RL, and chain-of-thought reasoning, but the core idea is explained without heavy mathematics.

Scope: One-sentence scope: the paper introduces Supervised Reinforcement Learning (SRL), a training framework that decomposes expert solution trajectories into step-wise "actions" and rewards a model by how similar its generated action is to the expert's action, enabling small open-source models to learn problems that neither SFT nor Reinforcement Learning with Verifiable Rewards (RLVR) can handle.

What This Paper Is About

Small open-source LLMs struggle on difficult multi-step reasoning problems where correct answers are almost never sampled, so reward-based RL gets no useful signal, while supervised fine-tuning on long expert demonstrations tends to overfit and can even degrade performance below the base model. SRL addresses this gap by treating problem solving as a sequence of logical actions: the model generates its own internal reasoning monologue and then commits to one action per step, receiving a dense, smoothly varying reward based on similarity to the corresponding expert action.

Key Contributions

1. The SRL framework itself: a reformulation of problem solving as sequential step-wise decision making with dense similarity-based rewards, designed for difficult problems where SFT and RLVR fail.
2. Empirical demonstration across two domains: challenging mathematical reasoning (AMC23, AIME24, AIME25, Minerva Math) and agentic software engineering (SWE-Bench-Verified), with SRL outperforming strong baselines in both.
3. An ablation study showing that guidance granularity matters: multi-step decomposition outperforms a one-step holistic sequence-similarity reward and a one-step final-answer reward.
4. Analysis of the reasoning behavior SRL induces, including interleaved planning, mid-solution adjustment, and reflective verification, which improve solution quality without simply lengthening outputs.

Main Findings

• SFT degrades or underperforms on hard data: On the s1K-1.1 dataset, the officially released S1K-7B model scored 24.1 greedy on AMC23 and 11.7 average, below the Qwen2.5-7B-Instruct base model, which scored 50.0 greedy on AMC23 and 24.6 average.
• RLVR only marginally improves: RL(VR) reached 47.5 greedy on AMC23 and 24.5 average, roughly matching the base model rather than improving on it.
• SRL substantially outperforms baselines: SRL scored 50.0 greedy on AMC23, 16.7 greedy on AIME24, 13.3 greedy on AIME25, 36.4 on Minerva Math, for a 27.6 average, described as a 3.0% average boost.
• SRL then RLVR is strongest: Training with SRL before refining with RLVR (SRL → RLVR) gave 57.5 greedy on AMC23, 20.0 greedy on AIME24, 10.0 greedy on AIME25, 36.4 on Minerva Math, for a 28.3 average, a 3.7% average increase using only 1k training data.
• SRL beats the reverse-curriculum baseline on hard benchmarks: R3 reached 10.0 greedy on AIME24 versus SRL's 16.7, and 7.4 greedy on AIME25 versus SRL's 13.3, though R3 was competitive on AMC23 at 57.5 greedy (equal to SRL → RLVR).
• Dynamic sampling helps SRL: Removing dynamic sampling dropped the average from 27.6 to 24.7, with AIME24 greedy falling from 16.7 to 13.3.
• Granular supervision beats holistic supervision: Multi-step SRL reached 27.6 average, versus 25.9 for a one-step sequence-similarity reward and 24.5 for a one-step final-answer reward (RLVR).
• The method scales down to a 3B model: With Qwen2.5-3B-Instruct, SRL → RLVR reached 19.5 average versus 16.4 for the base model and 17.7 for RL(VR), a 3.1% average improvement over the base model, with the largest gains on AMC23 (47.5 greedy) and Minerva Math (28.3).
• It transfers to software engineering: On SWE-Bench-Verified with Qwen2.5-Coder-7B-Instruct, SRL achieved 14.8 oracle file edit and 8.6 end-to-end resolve rates, versus 8.4 and 4.2 for SWE-Gym-7B and 5.8 and 3.2 for the base model. This is described as a 74% relative improvement over SWE-Gym-7B in the oracle setting and roughly double in the end-to-end setting.
• SRL changes reasoning style, not just accuracy: Models trained with SRL, particularly with RLVR, interleave reasoning throughout generation, showing structured planning, dynamic adjustment, and reflective verification instead of one monolithic reasoning block up front.

Methodology in Plain English

The researchers start from expert solution trajectories that already come with numbered steps. They break each solution into a sequence of actions, where one action is one logical step (an algebraic manipulation in math, or a command executed in a code repository for software tasks).

From a solution with N steps, they build N-1 training examples. For each example, the input is the problem plus all preceding steps, and the target is the next step. The model is prompted to write an internal monologue inside think tags and then produce the action for that step.

Instead of checking only the final answer, the training reward measures how similar the generated action is to the expert action, using a sequence similarity ratio computed from the longest matching blocks between the two texts (implemented with Python's difflib.SequenceMatcher). A format violation receives a reward of -1. Crucially, the reward is computed only on the external action, not the model's inner monologue, so the model is free to develop its own reasoning style while its actions stay aligned with expert strategy.

Training uses the GRPO objective with this dense reward. Because the reward varies continuously rather than being right-or-wrong, the authors generalize dynamic sampling: they filter out samples whose rollout rewards have near-zero standard deviation, since those provide weak learning signal, and keep sampling until the batch is full.

Evaluation uses greedy sampling accuracy on AMC23, AIME24, AIME25, and Minerva Math, plus average@32 at temperature 1.0 for the three competition benchmarks. Models are trained for up to 30 epochs, with the best validation checkpoint selected. For software engineering, they fine-tune Qwen2.5-Coder-7B-Instruct on 5,000 verified expert trajectories generated by claude-3-7-sonnet-20250219, producing 134k step-wise training instances and a 650-instance validation set, and evaluate both with oracle files and end-to-end with the Agentless-mini scaffold.

Why This Matters

The paper targets a practical bottleneck: outcome-based RL only works if the model can occasionally stumble onto a correct answer, and supervised fine-tuning on hard demonstrations can make small models worse. SRL offers a third path that produces useful learning signal even when every rollout is wrong, which matters for anyone training small models on data that is too hard for them.

Real-world applications:

• Training smaller, cheaper open-source models to handle competition-level math and multi-step quantitative reasoning without relying on a larger teacher at inference time.
• Building software engineering agents that edit real code repositories, using SRL to learn from expert agent trajectories where online RL is impractical.
• Domains with structured expert procedures, such as tool-use pipelines or other step-decomposable workflows, where an "action" can be defined cleanly enough to score by similarity.
• Curriculum-based training pipelines, since SRL works well as an initialization stage before RLVR refinement.

Industry relevance: The work comes from Google Cloud AI Research with co-authors at Google Cloud and UCLA, and it addresses a cost-sensitive question for practitioners: how to get the most reasoning capability out of a small base model and a modest amount of training data (as little as 1k examples in the math experiments).

Future Directions

• Extending SRL to domains where the notion of an "action" is ambiguous and reasoning cannot be cleanly partitioned into discrete steps.
• Reducing the dependence on highly structured, high-quality expert trajectories, which the authors identify as the method's primary constraint.
• Studying how different choices of similarity function affect performance and variance, which the authors list as an open question.
• Addressing the requirement that the student model already have baseline instruction-following ability, since without it the reward signal can become too sparse.

Target Audience

Researchers and engineers working on LLM post-training, reasoning, and agentic systems will benefit most, particularly those training small open-source models on difficult data where standard SFT and RLVR underperform. The paper is also useful for practitioners building software engineering agents, and for readers interested in the design space between imitation learning and reinforcement learning. Readers should be comfortable with concepts such as supervised fine-tuning, GRPO-style policy optimization, and chain-of-thought reasoning.

Authors’ abstract

Large Language Models (LLMs) often struggle with problems that require multi-step reasoning. For small-scale open-source models, Reinforcement Learning with Verifiable Rewards (RLVR) fails when correct solutions are rarely sampled even after many attempts, while Supervised Fine-Tuning (SFT) tends to overfit long demonstrations through rigid token-by-token imitation. To address this gap, we propose Supervised Reinforcement Learning (SRL), a framework that reformulates problem solving as generating a sequence of logical "actions". SRL trains the model to generate an internal reasoning monologue before committing to each action. It provides smoother rewards based on the similarity between the model's actions and expert actions extracted from the SFT dataset in a step-wise manner. This supervision offers richer learning signals even when all rollouts are incorrect, while encouraging flexible reasoning guided by expert demonstrations. As a result, SRL enables small models to learn challenging problems previously unlearnable by SFT or RLVR. Moreover, initializing training with SRL before refining with RLVR yields the strongest overall performance. Beyond reasoning benchmarks, SRL generalizes effectively to agentic software engineering tasks, establishing it as a robust and versatile training framework for reasoning-oriented LLMs.