SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning

AI summary

Overview

Research area: Post-training of large language models — reinforcement learning, self-distillation, and credit assignment for long-horizon reasoning and agentic tasks.

Technical level: Advanced. The paper assumes familiarity with policy gradient methods (PPO/GRPO), KL divergence, on-policy vs. off-policy learning, and behavior cloning.

Scope: The paper proposes SRPO, a two-stage framework that turns a model's own hindsight reflections into dense token-level training signals via on-policy self-distillation, and evaluates it on mathematical reasoning and long-horizon agentic benchmarks across four base models.

What This Paper Is About

Reinforcement learning post-training for LLMs relies on sparse terminal rewards — a single success or failure signal at the end of an episode — which gives only O(1) bits of information per episode and makes it hard to tell which of dozens or hundreds of intermediate decisions actually mattered. The paper's goal is to convert those sparse outcome signals into dense, token-level supervision by letting the model reflect on its own completed trajectories and then distilling that reflection-conditioned behavior back into the base policy. The result is a training loop that requires no external critic, no separate reward model, and no larger teacher model.

Key Contributions

1. Self-reflection as dense reward generation. SRPO reframes self-reflection not as an inference-time prompting trick but as a mechanism for producing dense supervision. Reflections on completed trajectories define a reflection-augmented "teacher" distribution, which is used to score the student's own on-policy rollouts token by token.

2. The "reset-with-memory" mechanism. Rather than iteratively appending reflections into an already-drifted context, SRPO prepends a compact reflection patch to the original prompt and regenerates from a clean initial state. Reflections are kept to roughly 2–5 bullet points.

3. On-policy self-distillation with a per-token reverse-KL estimator. The student minimizes the reverse KL divergence to the reflection-conditioned teacher under teacher-forcing on the student's tokens, using a cached single-sample Monte-Carlo log-ratio reward with a group-relative (trajectory-level) advantage baseline and a PPO-style clipped objective.

4. An empirical demonstration that self-distillation can replace larger teachers. Across model scales, SRPO with a self-teacher matches or exceeds distillation from substantially larger external models while using fewer teacher FLOPs.

Main Findings

• AIME'24 performance with the Qwen3-8B base model: 73.3%, which the paper reports as a 3.3 percentage point improvement over standard on-policy distillation and 5.3 points over GRPO.

• Data efficiency: SRPO reaches this 73.3% using only 8% (0.08×) of the training FLOPs required by scaled supervised fine-tuning.

• Compute versus GRPO: Counting Stage 1 and Stage 2, SRPO uses 5.4 × 10^18 FLOPs versus 20.8 × 10^18 for GRPO, i.e., approximately 3.8 times fewer total FLOPs.

• Agentic benchmark success rates: WebShop 64.7% (+7.9% over SFT), ALFWorld 76.8% (+5.6% over Reflexion), and SWE-Bench-Lite 31.2% (+4.4% over Reflexion).

• Shorter episodes: SRPO also achieves the shortest average episode length at 10.2 steps.

• Out-of-distribution generalization: On DeepScaleR, SRPO shows +7.8% relative improvement over on-policy distillation.

• Inference-time scaling: Each additional refinement iteration yields +1.8% on average for SRPO, versus +1.2% for GRPO and +0.9% for SFT. At 3 refinement iterations, SRPO reaches 78.5% on AIME'24, approaching models 4× larger.

• LoRA efficiency: LoRA-128 achieves 97.8% of full fine-tuning performance on AIME'24 while using only 1.3% of trainable parameters and 34% of the GPU memory (a 72–73% reduction). The paper notes that for standard SFT, LoRA lags by 13% after large-scale training.

• Scaling across model sizes (AIME'24): Qwen3-1.5B — SFT 38.2, GRPO 44.5, SRPO 52.3 (+7.8 over GRPO); Qwen3-8B — 60.0, 68.0, 73.3 (+5.3); Qwen3-32B — 71.4, 78.2, 82.0 (+3.8). Gains are largest for the smallest model.

• Compute Pareto efficiency: SRPO reaches 70% AIME'24 performance with roughly 10× fewer FLOPs than GRPO and 30× fewer than continued SFT.

• Self-teacher beats larger teachers (Table 3, Qwen3-8B student): GRPO 68.0 (0 teacher FLOPs), Qwen3-32B teacher 70.0 (4.0× FLOPs), Qwen3-72B teacher 72.5 (9.0× FLOPs), Self (SRPO) 73.3 (1.0× FLOPs). For a Qwen3-1.5B student: Qwen3-8B teacher 48.7 (5.3×), Qwen3-32B teacher 51.2 (21.3×), Self (SRPO) 52.3. SRPO outperforms distillation from Qwen3-72B by 0.8 points while using 9× fewer teacher FLOPs.

• Ablations (AIME'24 / WebShop): Full SRPO 73.3 / 64.7. Removing reflection (direct retry) 65.8 / 54.2 (a 7.5-point drop on AIME'24). Verbose reflection with more than 10 points 70.0 / 60.3 (−3.3 points). Outcome-only feedback 67.2 / 56.8. External teacher reflection 71.5 / 62.4 (−1.8 points). Forward KL instead of reverse KL 69.4 / 58.6 (a 3.9-point drop). Off-policy teacher trajectories 68.0 / 55.9. No clipping 70.2 / 61.3. Single-sample advantage without group normalization 71.1 / 62.0. Appending reflection rather than prepending 68.5 / 57.4. No state reset 66.3 / 52.8 (7.0 points worse than full SRPO).

• Reflection quality: Across 500 randomly sampled reflection instances from the AIME'24 training set rated by GPT-4 on a 1–5 helpfulness scale, 67% scored ≥4 and only 8% scored ≤2. Correlation between reflection helpfulness and rethinking-rollout improvement is r = 0.72. Score-5 reflections led to 34% average improvement in success rate; score 1–2 reflections yielded only 5%.

• Reflection failure modes: Generic advice 42%, incorrect diagnosis 35%, beyond capability 23%.

• Continual learning (Table 4): After training on math then adapting to code (SWE-Lite): SFT 60.0 before, 28.4 code after, 48.2 math after, 80.3% retention; GRPO 68.0, 26.7, 59.3, 87.2%; SRPO 73.3, 31.2, 69.8, 95.2%.

• Not reported in the available content: Specific numerical results for MATH-500, GSM8K, and AIME'24 under the Llama-3.1-8B-Instruct setting are not given in the provided text (the paper points to Figure 2 and Table 5). The continual-learning setup description is truncated mid-sentence.

Methodology in Plain English

SRPO runs in two stages using the same model, which is both student and teacher.

Stage 1 — reflect and rebuild the prompt. For each prompt, the model produces an initial attempt. In agentic settings that attempt is executed in an environment to get an outcome signal (success/failure, environment feedback, or a sparse scalar reward). The model is then asked to reflect on the tuple of (prompt, trajectory, outcome) and produce a compact "reflection patch" of roughly two to five bullet points — a diagnostic analysis plus actionable guidance. Crucially, instead of tacking that reflection onto a context that has already drifted, SRPO throws away the attempted context and rebuilds a clean initial state by prepending the patch to the original prompt. This means the environment state is untouched and the augmented prompt stays close to the original. The paper reports that a rollout sampled from this reconstructed state tends to be substantially higher quality, which is the evidence that it constitutes a valid teacher.

Stage 2 — distill the teacher back in, on the student's own tokens. The student, seeing only the original prompt, generates on-policy rollouts. For every token in those rollouts, the system computes the log-probability under the reflection-augmented teacher (teacher-forced on the student's own prefix) minus the log-probability under the cached behavior policy. This log-ratio is a single-sample Monte-Carlo estimator of the reverse KL divergence at that state, and it is the per-token reward. The paper describes the signal as "hindsight-guided dense supervision" — reflections are generated after seeing the whole trajectory, so the supervision densifies every token rather than performing precise per-step causal attribution. Advantages subtract a trajectory-level mean over non-padding token positions (a group-relative baseline inspired by GRPO, requiring no value network), and the update uses a PPO-style clipped objective. Because reverse KL is mode-seeking, the student is pushed to commit to the teacher's preferred behavior rather than spreading probability mass.

A key asymmetry results: training uses reflection, inference does not. The student learns to internalize behavior that was originally informed by outcome feedback, then reproduces it without the reflection prompt.

Experimental setup. Base models are Qwen3-1.5B, Qwen3-8B, and Qwen3-32B, plus Llama-3.1-8B-Instruct for cross-family generalization on agentic tasks. Math training starts from an SFT checkpoint trained on 400K prompts from OpenThoughts-3, with batch size 256 rollouts, learning rate 1×10^-5 with cosine decay, and clip ratio ε = 0.2. Agentic training uses 64 prompts per batch with 4 samples per prompt. All experiments use 8× H100 GPUs unless otherwise specified. Benchmarks: AIME'24 (30 competition-level problems), MATH-500 (500 problems across 7 categories), GSM8K (1,319 test problems), DeepScaleR (1,200 challenging problems), WebShop (12,087 shopping tasks), ALFWorld (134 household tasks), and SWE-Bench-Lite (300 real-world GitHub issues). Baselines include SFT, GRPO, PPO, on-policy distillation from Qwen3-32B, Reflexion (inference-time only), and Self-Refine.

Why This Matters

Impact on research. The paper attacks the credit assignment bottleneck directly, arguing that distillation offers O(T) bits of supervision per episode where sparse-reward RL offers only O(1). It also questions a default assumption in distillation work — that a teacher must be larger and more capable — by showing that the same model operating under hindsight information can serve as its own teacher. The reported results against reflection-trained baselines (SCoRe, R3L, RISE) and against on-policy distillation from a 72B external teacher suggest a different route to scaling post-training than simply scaling teachers.

Real-world applications:

• Long-horizon agents: web navigation, shopping, and household task assistants where episodes span roughly 10–50 actions and a single terminal reward cannot identify which step failed.
• Software engineering agents: resolving real GitHub issues, where the SWE-Bench-Lite result of 31.2% success is directly relevant.
• Low-resource post-training: the LoRA-128 result (97.8% of full fine-tuning accuracy at 1.3% of trainable parameters and 34% of GPU memory) and the elimination of larger teacher models lower the compute barrier to strong post-training.
• Continual skill acquisition: the 95.2% retention reported in the math-then-code setting speaks to production systems that must add capabilities without regressing on existing ones.

Industry relevance. The 3.8× FLOPs reduction versus GRPO, the roughly 10× FLOPs advantage over GRPO to reach 70% AIME'24 performance, and the fact that practitioners avoid API calls to larger teacher models all translate directly into training cost. The compute-efficiency and memory-efficiency claims are the kind of numbers that determine whether a post-training recipe is deployable at scale. Code is available at https://github.com/Galleons2029/SRPO.

Future Directions

1. Improving weak reflections. The failure-mode breakdown — 42% generic advice, 35% incorrect diagnosis, 23% beyond capability — is the paper's own identified ceiling. The authors suggest incorporating external verification signals or retrieval-augmented reflection as remedies.

2. Extending beyond the benchmarks tested. The continual-learning section is included, but the provided setup description is truncated; a fuller treatment of sequential skill acquisition and forgetting across more than two task families is a natural extension.

3. Understanding when self-teaching fails. Since 23% of unsuccessful reflections arise from problems the model lacks knowledge for, an open question is how SRPO behaves on domains where the base model has no latent capability to "unlock."

4. Interaction with other inference-time and training-time methods. The paper reports better inference-time scaling for SRPO-trained models (+1.8%/iter), which raises the question of how far that synergy extends when combined with other refinement or search procedures.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for language models, and agentic systems. It will be most useful to readers already comfortable with PPO/GRPO-style objectives, KL-based distillation, and on-policy versus off-policy distinctions, since the method's core argument rests on which divergence is used and where the gradient flows. Practitioners with constrained compute — those deciding between LoRA and full fine-tuning, or between training their own model and distilling from a larger one — will find the efficiency tables directly actionable. Readers looking for an introduction to RL post-training will find the paper assumes too much background to serve as a starting point.

Authors’ abstract

Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with exceptional data efficiency. Using a Qwen3-8B base model, SRPO attains 73.3% on AIME'24 using only 8% (0.08x) of the training FLOPs required by scaled supervised fine-tuning, while significantly improving success rates on WebShop (64.7%), ALFWorld (76.8%), and SWE-Bench-Lite (31.2%). Code is available at https://github.com/Galleons2029/SRPO