SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning

AI summary

Overview

• Research area: Reinforcement learning for large language model agents that act over many turns in interactive environments (agentic RL, on-policy distillation, credit assignment).
• Technical level: Advanced. The paper assumes familiarity with group-based policy optimization (GRPO/RLOO-style objectives), clipped policy ratios, and on-policy distillation.
• Scope: The paper introduces SERO, a training-time auxiliary objective that restores a state visited by the current policy, lets a privileged oracle act in the real environment for multiple steps, and trains the policy on those executed actions with reachability-based weighting; it is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct.

What This Paper Is About

Group-based agentic RL methods assign credit using only the trajectories the current policy happens to sample, so if every rollout in a group misses a useful action sequence, no better credit estimator can supply that missing behavior as a positive target. Recent teacher-guidance methods such as on-policy distillation query a teacher on learner-visited states, but their supervision is centered on a single corrective action, a teacher prefix, or the outcome of an intervention rather than the full missing branch. SERO closes this gap by turning an oracle correction into a multi-step, environment-executed training trajectory whose downstream observations and oracle actions are themselves learned.

Key Contributions

1. Executable oracle continuations. SERO restores a simulator snapshot for a state actually reached by the current policy, then lets a training-only oracle interact with the real environment for multiple steps. Because every oracle action is executed, each downstream target is conditioned on the observation produced by the preceding oracle action, so the method learns the consequences of an intervention rather than just the intervention itself.
2. Policy-reachability weighting. Each oracle step is weighted by the product of length-normalized behavior-policy compatibility scores over the preceding oracle actions. The root action always gets unit prefix weight, and deeper targets are down-weighted when the policy cannot plausibly follow the oracle prefix that leads to them, so the effective supervision horizon extends as the policy improves.
3. A branch-level improvement gate. A detached gate compares the realized return of the oracle continuation against the on-policy suffix from the same restored root over the same remaining horizon, and clips the normalized difference into [0, 1], controlling whether and how strongly a branch contributes.
4. A modular auxiliary objective. The oracle loss is added alongside the base RL loss with weight λ_O, leaving the base method's rewards, group statistics, advantages, and importance ratios untouched; no oracle or environment restoration is used at inference. It is demonstrated on top of both GraphGPO and HGPO.

Main Findings

• SERO improves both backbones at both scales. At 1.5B, GraphGPO+SERO raises ALFWorld in-distribution/out-of-distribution success from 90.63/88.02 to 95.83/91.67 and WebShop task score/task success from 87.27/77.93 to 88.90/79.27. HGPO+SERO raises ALFWorld from 92.19/87.24 to 94.53/90.10 and WebShop task success from 73.70 to 77.47 (task score 87.80 to 88.43).
• Largest reported gain is on out-of-distribution evaluation. At 7B, GraphGPO+SERO moves ALFWorld from 91.67/76.30 to 95.57/87.50, an improvement of 11.20 points on unseen tasks, while WebShop goes from 89.77/80.47 to 91.93/83.67. HGPO+SERO at 7B moves ALFWorld from 94.01/90.89 to 94.53/91.41 and WebShop task success from 74.47 to 79.63.
• Root-only oracle supervision does not help. Supervising only the first oracle action and discarding the continuation decreases ALFWorld seen/unseen success relative to GraphGPO by 2.87/4.17 points (to 87.76/83.85) and barely changes WebShop (87.30/77.73).
• Full continuations help; flat weighting helps less. With a shared branch-level weight (Full-Flat), ALFWorld reaches 94.79/88.28 and WebShop 87.63/78.33. Relative to Root-only, learning the full executable continuation adds 7.03/4.43 points on ALFWorld and 0.33/0.60 on WebShop.
• Reachability weighting adds further gains, mostly out-of-distribution. Compared with Full-Flat, the exclusive-prefix weighting in SERO adds 1.04 points on ALFWorld seen success and 3.39 on unseen success, plus 1.27 task score and 0.94 task success on WebShop.
• Fixed-bank control supports the weighting rule itself. Replaying all weighting variants on the same frozen oracle bank (271 valid roots, 5,002 oracle-continuation steps, built from the 1.5B GraphGPO checkpoint at step 50), SERO reaches 88.02/85.94 seen/unseen versus 85.42/84.64 for flat within-branch weighting and 87.24/82.55 for a depth-decay baseline.
• Local compatibility alone is insufficient. The per-step score z_k fluctuates and often stays high even for late oracle actions, whereas the prefix product w_k drops quickly once an early oracle action is poorly supported. At training step 10 the prefix weights decay early, while at step 140 they stay large over a longer portion of the continuation.
• Moderate oracle-loss weights are stable. Performance holds for λ_O in [0.1, 0.5], degrades at λ_O = 1.0, and is weaker at λ_O = 0.05, particularly on unseen ALFWorld tasks; λ_O = 0.1 is the default.
• SERO also outperforms standard baselines. At 1.5B, GRPO reaches 72.8/70.1 on ALFWorld and 75.8/56.8 on WebShop, PPO 54.4/–/73.8/51.5, RLOO 69.7/68.7/73.9/52.1, and GiGPO 88.27/86.47/83.10/65.00. Prompting baselines are far lower (for example, ReAct at 1.5B: 12.8 ALFWorld seen, 40.1 WebShop score, 11.3 task success). Closed-source prompting references are GPT-4o at 48.0/46.0/31.8/23.7 and Gemini-2.5-Pro at 60.3/50.5/42.5/35.9.

Methodology in Plain English

The training loop has three stages per iteration. First, an ordinary group-based RL update runs as usual: the policy samples several trajectories per task, and the base method computes its own advantages and clipped objective from those rollouts only. Second, SERO picks a pre-action state from an on-policy trajectory — specifically from a group that contains at least one failed rollout — restores the saved simulator snapshot in a separate environment copy, and lets a training-only oracle act from there until the task finishes or the remaining interaction budget runs out. Because each oracle action is actually executed, the oracle at step k sees the observation created by its own earlier actions, giving a valid multi-step target sequence.

Third, the policy is trained to reproduce those oracle actions with a supervised likelihood loss, scaled by two detached factors. The first is an improvement gate: if the oracle continuation's realized return beats the policy's own suffix return from the same root by enough (normalized by a scale c_R, which is 1 for binary success), the branch contributes; otherwise it is suppressed. The second is the reachability weight: the behavior policy's length-normalized likelihood of each oracle command is computed at each oracle state, and the weight for step k is the product of these scores over all earlier steps. The root gets weight 1; deeper steps get progressively smaller weights when the policy would struggle to follow the prefix leading there. Both factors are stop-gradient, so they act as fixed scalars on the loss.

The total objective is the base RL loss plus λ_O times this oracle loss, with λ_O = 0.1 by default. The trainable policy only ever sees the policy-visible observation at each oracle state; the oracle may use privileged information such as the ALFWorld expert planner or a WebShop goal-metadata oracle, and this machinery is removed at inference. Ablations isolate each design decision: Root-only (first oracle action only), Full-Flat (whole continuation with one averaged branch weight, constructed to preserve the total branch weight), and full SERO with per-step exclusive-prefix weights. Results are averaged over 3 random seeds during testing.

Why This Matters

• Research impact: The paper reframes teacher guidance for agents as learning an executable future rather than a single corrective action, and shows empirically that the multi-step version is what produces gains. The reachability weighting offers a general recipe for deciding how far down an off-policy branch it is safe to supervise.
• Compatibility: Because SERO is an additive, decoupled loss that leaves advantages and importance ratios unchanged, it can be dropped onto existing group-based agent trainers, which the paper demonstrates with both GraphGPO and HGPO.
• Real-world applications:
  • Web navigation and e-commerce agents that must search, filter attributes, and complete a purchase.
  • Household or embodied instruction-following agents operating in simulated environments.
  • Tool-use and software-task agents where a wrong intermediate call changes all subsequent observations.
  • Any multi-turn agent setting where a simulator or privileged metadata source exists at training time but not at deployment.
• Industry relevance: The method needs no oracle at inference, so deployment cost is unchanged, and the strongest reported gains are on out-of-distribution tasks — the setting that matters most when agents face unseen instructions after deployment.

Future Directions

• Reducing oracle dependence. Both benchmarks rely on privileged training-time sources (an ALFWorld expert planner and a WebShop goal-metadata oracle); how SERO behaves with weaker, noisier, or self-generated oracles is not established here.
• Root selection and budget. The paper uses one root per eligible group sampled from groups containing at least one failure; how root choice, multiple roots per group, or longer continuations affect the trade-off between supervision depth and off-policy drift remains open.
• Beyond binary returns. The improvement gate uses c_R = 1 for binary success returns; how to normalize it for dense or multi-objective rewards is left open.
• Interaction with finer-grained credit assignment. SERO is shown on top of GraphGPO and HGPO, and the paper frames it as compatible with GRPO, GiGPO, HGPO, and GraphGPO, but combinations with newer hindsight or uncertainty-guided estimators are not tested.

Target Audience

Researchers and engineers working on reinforcement learning for LLM-based agents, particularly those already using group-based critic-free methods (GRPO, RLOO, GiGPO, GraphGPO, HGPO) on multi-turn interactive benchmarks. It is also relevant to practitioners of on-policy distillation and imitation learning who care about how far off-policy a supervision signal can safely reach, and to teams building web, tool-use, or embodied agents who have access to a simulator or privileged annotation during training only.

Authors’ abstract

Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce \textbf{State-Rooted Executable Rollout Optimization (SERO)}, which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies \textbf{policy-reachability weighting}, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.