ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning

AI summary

Overview

• Research area: Post-training of large language models for reasoning, specifically on-policy training paradigms such as Reinforcement Learning with Verifiable Rewards (RLVR) and On-Policy Distillation (OPD).
• Technical level: Intermediate — the paper builds on GRPO-style group-relative objectives and reverse-KL distillation, but its core idea (learn from a strong model's failures by reflecting on them) is described in largely conceptual terms.
• Scope in one sentence: ReflectRL is a lightweight, plug-and-play framework that turns expert-model failure trajectories into training signal by having a policy reflect on them, then gradually transfers that reflective behavior back into ordinary direct reasoning.

What This Paper Is About

Trajectory-guided on-policy training usually relies on a stronger expert model (for example DeepSeek-R1) to supply correct "golden" reasoning trajectories as supervision. When the expert fails on a harder problem, that supervision disappears and the failed trajectory is normally discarded as a negative sample. The paper argues these failures — which the authors call Golden Negative Trajectories (GNTs) — still carry useful signal if they are treated as flawed trajectories to reflect upon rather than demonstrations to imitate. The goal is a method that exploits GNTs without changing the underlying training objective, the verifier, or the inference-time interface.

Key Contributions

1. Introduces Golden Negative Trajectories as a reasoning signal for on-policy training, and releases OpenR1-GNT-69k, a dataset of 69k expert failure trajectories.
2. Identifies the Reflection Advantage, the empirical pattern that hard problems can be easier and more effective to solve through reflective reasoning than through direct reasoning.
3. Proposes ReflectRL, a lightweight framework that integrates GNTs into on-policy training via Reflective-to-Direct Policy Transition.
4. Demonstrates consistent improvements across representative on-policy training settings covering RLVR and OPD, with minimal additional overhead.

Main Findings

• Reflection Advantage is real and grows: On Qwen2.5-Math-7B, reflecting on a GNT produced the largest accuracy gain (Reflection Gain Δ_ref, the expected correctness difference between reflective and direct reasoning), and this gain steadily increased during training. Self-failures offered limited benefit, while weak-model failures or a length-matched mismatched GNT produced negative gain.
• GNTs are structurally higher quality: Using Qwen2.5-Math-PRM-7B score profiles, GNTs maintained a higher valid reasoning prefix than other failure sources before dropping at localized error points.
• Both parts of a GNT matter causally: Ablating GNT components showed the valid prefix and the error region jointly drive the reflection gain, while shuffled GNTs and answer-only variants dropped accuracy below the direct baseline.
• Consistent gains across methods: ReflectRL improved GRPO from an in-distribution average of 37.0 to 42.4 (+5.4) and an out-of-distribution average of 20.9 to 40.0 (+19.1); DAPO from 38.4 to 43.5 (+5.1) in-distribution and 25.3 to 37.0 (+11.7) out-of-distribution; and OPD from 26.7 to 29.0 (+2.3) in-distribution and 21.6 to 39.1 (+17.5) out-of-distribution.
• Correct expert trajectories are not automatically better guidance: Correct expert trajectories (GT guidance) gave an early improvement but produced severe policy drift and eventual collapse, whereas GNT guidance remained stable. The figure highlights steps 70–130 as the region linking severe policy drift to collapse.
• Shorter, more efficient reasoning: GRPO produced responses exceeding 800 tokens by step 500, while ReflectRL stayed near 420 tokens while achieving higher accuracy.
• Less entropy collapse: GRPO entropy fell from 0.97 to below 0.03 by step 250, while ReflectRL remained around 0.15 at the same step, roughly five times higher.
• Lower per-step training cost: ReflectRL stabilized at approximately 13 seconds per update step versus around 20 seconds for GRPO.
• Smooth transitions win: The cosine kernel achieved the best final accuracy, with the inverse-sigmoid variant following a similar trend; sharper transitions weakened performance by removing reflective guidance before the policy had internalized the behavior.
• Generalizes across models: Improvements held across Llama-3.1-8B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen2.5-3B-Instruct on the 9 in-distribution and out-of-distribution benchmarks.
• Transfers beyond math: OpenR1-GNT-69k and ARC-c have no data overlap, so the ARC-c gains are attributed to transfer beyond the mathematical training distribution.

Methodology in Plain English

The approach rests on two prompt templates applied to the same policy. Chat_Temp_D gives the model only the original question (direct reasoning, the default mode in on-policy training). Chat_Temp_R gives the question plus a pre-generated GNT and asks the model to identify the errors, repair the reasoning, and derive a corrected solution (reflective reasoning).

During RLVR training, each query generates a mixed group of N rollouts: the first K_t rollouts use the reflective template and the rest use the direct template. All rollouts are scored by the same verifier (MathVerify) and advantages are computed jointly over the mixed group with the standard group-normalized (GRPO-style) estimator, so no auxiliary loss term is introduced. Because rewards are binary, group normalization only changes advantage magnitude — correct rollouts keep positive advantages and incorrect ones keep negative advantages regardless of interface, so direct reasoning is not systematically disadvantaged.

To avoid the model becoming dependent on GNTs that are unavailable at inference, the proportion of reflective rollouts follows a transition kernel g(t), by default cosine decay from an initial proportion p_h to a terminal proportion p_l, with K_t = round(N · g(t)). In the RLVR setting the terminal allocation rounds to zero, yielding fully direct rollout groups.

For OPD, the GNT is supplied only on the teacher side as privileged context; the student generates rollouts and is always aligned with the direct interface, matching a reverse-KL target produced by the teacher that has reflected on the error prefix. This way the student never observes the GNT and distills error-avoidance and correction behavior into its direct pathway.

Implementation details: the work is implemented in verl; rollout batch size 128, update batch size 64, 8 sampled responses per on-policy question, evaluation temperature 0.6, OPD uses reverse KL with a rollout batch size of 1024, and all experiments run on 16 H200 GPUs. The GNTs are pre-generated offline (from a subset of OpenR1-Math-220k, with prompts from NuminaMath 1.5 and candidate trajectories from DeepSeek-R1, retaining those verified incorrect by Math-Verify), so no online expert queries are required, and rollout and verifier budgets match the baselines.

Why This Matters

• Research impact: The work reframes discarded expert failures as a first-class supervision source and shows that correct expert trajectories can actively destabilize training while flawed ones can stabilize it. It offers a way to extend trajectory-guided RLVR and OPD to problems where the expert never succeeds, without new loss terms, rewards, or inference interfaces.
• Real-world applications:
  • Training reasoning models in domains where even the strongest available expert fails frequently, so correctness-filtered pipelines yield little signal.
  • Cost-efficient post-training: reusing offline expert failures avoids additional online expert queries and reduces per-step training time.
  • Building models that diagnose and repair their own mistakes, useful for tutoring or error-explanation systems.
  • Transfer to out-of-distribution reasoning tasks such as science and logic multiple-choice benchmarks, as seen in the ARC-c results.
• Industry relevance: Because ReflectRL is plug-and-play and adds no auxiliary model, trainable parameters, gradients, or optimizer states (its only extra cost is a temporary KV cache for the reflective context), it can be dropped into existing RLVR or distillation codebases with unchanged rollout and verifier budgets.

Future Directions

• Scaling the GNT corpus: OpenR1-GNT-69k covers a specific pipeline; whether larger or more diverse failure sets from other experts and domains produce greater Reflection Advantage is not established.
• Beyond math and multiple-choice transfer: The demonstrated transfer is to ARC-c, GPQA-Diamond, and MMLU-Pro; whether the effect holds for open-ended generation, code, or agentic tasks is not reported.
• Choosing and adapting transition schedules: The paper compares six kernels and finds smooth cosine decay best, but the functional form matters less than gradualness; adaptive or problem-dependent schedules remain unexplored.
• Diagnosing the GT-guidance collapse: The causal link between policy drift and collapse under correct expert guidance is shown but not fully explained, leaving open how to combine correct and negative expert trajectories safely.

Target Audience

Researchers and practitioners working on LLM post-training, reinforcement learning with verifiable rewards, and on-policy distillation; engineers who need a low-overhead way to extract training signal from expert model failures; and readers interested in self-correction, error repair, and reasoning-efficiency behaviors in large language models.

Authors’ abstract

On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.