Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

AI summary

Overview

Research area: Reinforcement learning for AI agents, specifically reinforcement learning with verifiable rewards (RLVR), plus self-distillation for agentic and tool-integrated reasoning.

Technical level: Advanced. The paper assumes familiarity with RLVR, group-relative policy optimization (the family that includes GRPO-style objectives), and self-distillation, and it reasons about where learning signal does and does not exist in those setups.

Scope: The paper proposes "prospective learning" and a concrete method, Self-Retrospection Distillation (SRD), which converts information available only after an agent finishes a trajectory into a training target for foresight predictions made before any interaction begins.

What This Paper Is About

Most agent training with verifiable rewards learns only from a scalar score received after the agent acts. A known failure mode of group-relative methods is that when every rollout in a group earns the same score, the relative comparison produces no gradient at all — the agent learns nothing, even though the trajectories themselves may show what the task demanded and how the agent went wrong. The paper asks whether that after-the-fact experience can instead teach an agent what it should have anticipated before it started, and it builds SRD to do exactly that. The goal is a complementary learning signal that keeps working when reward-based contrast disappears.

Key Contributions

1. Introduces "prospective learning" — a framing in which post-hoc experience supervises predictions about a task made from the pre-interaction, trajectory-blind view, rather than only supervising the actions taken during interaction.
2. Proposes Self-Retrospection Distillation (SRD) — an instantiation in which a completed trajectory's privileged hindsight is distilled into the same policy's foresight, so the policy learns to anticipate what the task requires and which pitfalls to avoid.
3. Demonstrates complementarity with existing methods — SRD is shown to add to RLVR and to self-distillation baselines rather than replacing them, with gains of up to 24.2 pp across 10 tool-integrated reasoning and long-horizon agentic tasks.
4. Shows the method's advantage is largest when reward contrast is scarce — including the extreme case where nearly all rollout groups are reward-uniform, and no overall inference-time cost is added because foresight is only a training target.

Main Findings

• Reward-uniform groups are common, not rare: between 37% and 98% of rollout groups are reward-uniform across model scales, meaning group-relative reward signal is frequently absent even though trajectories carry information.
• SRD recovers learning signal from those trajectories: it exploits sampled trajectories that scalar-reward methods discard, which the abstract presents as the reason its advantage is most pronounced when contrast is scarce.
• The most dramatic reported case is a 2B model: with 98% of groups being all-failure, RLVR training ends at 0.0% success, while adding SRD reaches 60.6% success under the same rollout budget.
• Gains generalize across task types: improvements of up to 24.2 pp are reported over RLVR and self-distillation baselines across 10 tool-integrated reasoning and long-horizon agentic tasks.
• No inference-time overhead by design: foresight is used only as a training target and does not need to be explicitly generated when the model is deployed.
• Broader claim: the authors argue post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before interaction is available.

Methodology in Plain English

The approach rests on a simple asymmetry between two views of a task:

• The hindsight view. After the agent has attempted a task, the full trajectory is known — what it did, where it failed, what the task apparently required. This is "privileged" information because it could not have been available beforehand.
• The foresight view. Before interacting, the agent sees only the task, not any trajectory. In this view it must predict what matters: what the task will demand and which mistakes to avoid.

SRD takes the useful knowledge implied by a finished trajectory and uses it as supervision for the policy's own foresight predictions from the trajectory-blind view. In effect, the same policy plays teacher and student: the completed attempt tells it what it should have seen coming. Because the target is only a training signal, the deployed model is unchanged — it does not have to output foresight at inference time. This makes the method orthogonal to RLVR, so the two signals can be combined: reward-based learning when reward contrast exists, and retrospective foresight distillation when it does not. The abstract does not describe the loss formulation, the exact form of the foresight prediction, or the training pipeline in detail; those specifics are not available from the abstract alone.

Why This Matters

Impact on research. The work names a failure mode of group-relative RLVR — zero learning signal under reward-uniform groups — and offers a signal source that does not depend on reward contrast. It reframes a trajectory as something more than a vehicle for an outcome reward: it is also evidence about the task. That reframing opens a different axis for improving agents than reward shaping, filtering, or better sampling.

Real-world applications (implied by the task settings the abstract names):

• Tool-integrated reasoning agents that must decide which tools to call and in what order, where many attempts can fail identically.
• Long-horizon agentic tasks with sparse success, where failure is the norm early in training and outcome rewards are nearly useless.
• Domains with expensive or binary verification, where a checker can say "wrong" but rarely provides a graded signal to learn from.
• Settings with limited rollout budgets, where the ability to extract training value from already-collected trajectories matters more than collecting more of them.

Industry relevance. The 2B-model result is the practically significant one: at that scale, 98% of rollout groups were all-failure and standard RLVR yielded 0.0% success, while adding SRD reached 60.6% under the same rollout budget. Successful learning from failure-heavy rollout batches, at small model sizes and without extra inference cost, is directly relevant to teams that cannot afford large-scale rollouts or large models.

Future Directions

• Characterizing when hindsight is trustworthy. If a trajectory fails entirely, what can it legitimately teach about what the task required? The abstract does not address how hindsight quality is judged or filtered.
• Interaction with other reward-free or sparse-reward methods. SRD is presented as complementary to RLVR and self-distillation; how it combines with other approaches to the reward-uniform problem is left open.
• Scaling and generality. Results span several model scales and 10 tasks, but the abstract does not establish how the method behaves on tasks outside tool-integrated reasoning and long-horizon agency.
• Form of the foresight target. Since foresight is never emitted at inference, it is an open question what the optimal predictive target is and whether richer hindsight (e.g., information beyond the trajectory itself) would help.

Target Audience

Researchers and practitioners working on reinforcement learning for language-model agents, especially those using group-relative objectives such as GRPO-style training and encountering stalled or zero-gradient updates from uniform rewards. It is also relevant to engineers building tool-using or long-horizon agents under tight rollout budgets, and to anyone interested in self-distillation methods that use a model's own completed experience as supervision. Readers without a background in RLVR and group-relative policy optimization will need that context first, given the paper's technical level.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to $24.2$ pp. Its advantage is especially pronounced when reward contrast is scarce: when $37$--$98\%$ of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In the 2B setting, where $98\%$ of groups are all-failure, the RLVR training ends up at $0.0\%$ success, while adding SRD reaches $60.6\%$ under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction.