RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

AI summary

Overview

Research area: Natural Language Processing (cs.CL), specifically agentic reinforcement learning and LLM post-training for long-horizon, multi-turn agents.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods (GRPO, PPO), on-policy distillation, reward normalization, and empirical Bayes estimation.

Scope: The paper introduces and evaluates RSIPO, a self-evolving agentic RL framework that co-evolves an LLM's skill-generation ability with its task-execution policy and calibrates how much the student trusts a skill-enhanced teacher.

What This Paper Is About

Existing hybrid agentic RL methods combine outcome-based reinforcement learning with on-policy self-distillation, where a teacher conditioned on agentic skills scores the student's tokens. The paper identifies two weaknesses: skills are usually retrieved from a static library or produced by an off-policy generator, and the teacher is assumed to be more reliable than the student even though skill access does not guarantee better decisions. RSIPO addresses both by measuring how much each self-generated skill actually improves execution through trajectory replay, then using those measured gains to train skill generation on-policy and to weight teacher supervision per task.

Key Contributions

1. Co-evolving perspective. The authors diagnose two limitations in prior agentic RL: skill generators are trained off-policy and do not evolve with the policy, and OPSD uniformly trusts teacher guidance even when the skill-enhanced teacher is less reliable than the student.

2. Execution check via trajectory replay. RSIPO splits rollouts into an evidence set and a control set, rewrites a skill from an evidence trajectory, replays the skill-enhanced teacher (K) times from the same initial state, and computes a replay gain relative to the control rollouts.

3. On-policy skill evolution. The replay gain, centered by the mean gain for the same task type, becomes a reward for the skill-generation tokens. This explicitly optimizes skill writing on-policy without auxiliary judge models or additional evaluation data.

4. Trust-calibrated self-distillation. Type-level empirical Bayes shrinkage converts noisy replay gains into robust per-task estimates, which are mapped to trust coefficients that upweight, downweight, or disable the OPSD loss. Experiments across Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-1.7B on ALFWorld and seven search datasets show state-of-the-art or near-state-of-the-art results without using skills at test time.

Main Findings

• Strong ALFWorld performance. RSIPO reaches average success rates of 95.7%, 98.6%, and 95.0% on ALFWorld with Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B, respectively. These are gains of 3.9%, 2.5%, and 3.0% over the strongest baseline, SEED, and 20.7%, 17.4%, and 48.9% over standard GRPO.

• Large gains over the naive hybrid. GRPO+OPSD underperforms GRPO with Qwen2.5-7B and Qwen3-1.7B, consistent with the paper's warning that uniformly trusting the teacher can hurt. RSIPO outperforms GRPO+OPSD by 14.5%, 18.2%, and 63.0% on the three backbones.

• Search-based results. RSIPO achieves the highest average exact-match accuracy with Qwen2.5-7B (50.8%) and Qwen3-1.7B (42.6%), and ranks second with Qwen2.5-3B (45.3%), 0.4% below SEED. It improves over GRPO by 8.9%, 8.8%, and 1.8% on the three backbones.

• No test-time skills required. Although RSIPO uses no agentic skills at inference, it outperforms baselines that do use test-time skills, including Skill-Prompt* and Skill-GRPO*. With Qwen2.5-7B it surpasses Skill-GRPO* by 10.3% on ALFWorld and 3.3% on search tasks.

• Evidence that skill generation evolves. The preliminary analysis shows only 0.25% of skill-generator SFT trajectories are successful, the base model follows the action format in just 11.0% of steps, and 86.8% of SFT skills concern action format or syntax, while 99.7% of the policy's actions are valid within the first 10 RL steps. After RSIPO training, Procedure-style skills exceed 94% in the last 45 steps and the replay gain turns positive.

• Teacher reliability is not guaranteed. The skill-enhanced teacher underperforms the student on 5.5% to 22.4% of tasks depending on student competence. In the first 15 RL steps the teacher underperforms on more tasks than it outperforms (22.9% versus 15.0%) and continues to underperform on 4% to 11% of tasks thereafter.

• Ablation supports both components. On ALFWorld with Qwen2.5-3B, OPSD alone raises the base model from 21.9% to 28.1%, GRPO reaches 75.0%, GRPO+OPSD reaches 81.2%, and SFT initialization brings the hybrid to 86.4%. Dynamic trust and skill evolution add 3.6% and 5.6%, respectively, reaching 95.7%; removing either component costs 3.7% to 5.7%.

• Trust calibration behaves as intended. On ALFWorld the dynamic trust exceeds 1.1 within 20 steps as the skill-enhanced teacher increasingly outperforms control rollouts. On search-based tasks, where skills add little, it stays close to one, degrading gracefully to approximately standard OPSD.

Methodology in Plain English

RSIPO builds on a hybrid agentic RL setup with two branches. The RLVR branch samples several rollouts per task and assigns a group-normalized outcome advantage to every token, as in GRPO. The OPSD branch lets the same model act as a teacher by inserting a self-generated skill into its context, then trains the student to reduce the gated log-probability gap on its own tokens.

The new ingredient is a replay-based execution check. For each training task, the rollouts are split into an evidence half and a control half. The policy rewrites a skill from one evidence trajectory, preferring a successful one. The skill-enhanced teacher is then replayed several times from the exact initial task state, and its success is scored with a utility that also rewards faster completion. The replay gain is the replay group's average utility minus the control group's average utility.

Because a handful of episodes gives a noisy gain, RSIPO pools gains by task type across training steps using moving averages, estimates the between-task variance, and applies empirical Bayes shrinkage. Each task's gain is pulled toward its type mean in proportion to how noisy it is. The shrunk gain is used twice. First, it becomes a centered reward for the tokens of the generated skill, so skill writing is optimized by policy gradient on-policy. Second, it is mapped through a clipped linear function to a trust coefficient that multiplies the OPSD loss for that task, letting the student downweight or entirely disable unreliable teacher guidance. All three losses are optimized together in a single update.

Experiments use ALFWorld for embodied interaction and seven search datasets following Search-R1: three single-hop datasets (NQ, TriviaQA, PopQA) and four multi-hop datasets (HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle). Training uses 150 steps, 8 rollouts per task, and 16 tasks per step on ALFWorld or 128 on search tasks, with a learning rate of (10^{-6}) on 8 NVIDIA H20 GPUs. Skills are initialized by supervised fine-tuning on trajectory-skill pairs annotated by GLM-5.2.

Why This Matters

Research impact. The paper reframes agentic skills as something to be optimized on-policy rather than retrieved or pre-trained once. It also challenges the common assumption that a privileged-context teacher is always more reliable, and it demonstrates a statistically principled way to handle noisy per-task reward signals inside an RL loop. The result is a stronger hybrid of RLVR and on-policy self-distillation without test-time skill prompting.

Real-world applications.

• Household and embodied assistants that must plan and execute multi-step manipulation tasks, as studied in ALFWorld.
• Search and retrieval-augmented agents that answer single-hop and multi-hop questions by issuing queries and reading evidence.
• Tool-use and workflow agents that need reusuable procedures rather than fixed prompt libraries.
• Future coding and co-working assistants, which the authors list as an extension target rather than a demonstrated result.

Industry relevance. Companies training long-horizon LLM agents can use RSIPO-style replay checks to turn their own rollouts into a skill-training signal without building a separate judge model or maintaining a static skill library. The trade-off is additional rollout compute and a requirement that the environment can be reset to a specified state, which matters for simulator-based and offline-retrieval deployments but limits live-web settings.

Future Directions

• Scaling to larger models and broader tasks. The authors explicitly plan to extend RSIPO to larger models and to coding and co-working scenarios.
• Reducing replay cost. The extra skill-enhanced replays improve training but increase compute; a cheaper or adaptive replay budget is an open engineering question.
• Non-resettable environments. RSIPO relies on resetting to the exact initial state, which is difficult for live websites and other irreproducible settings.
• Better statistical calibration. The shrinkage and trust mapping use task-type statistics and fixed hyperparameters ((\kappa = 2), (\omega_{\max} = 2)); whether these transfer across task distributions or could be learned is not resolved.

Target Audience

This paper is best suited to researchers and graduate students working on LLM post-training, agentic reinforcement learning, and on-policy distillation, as well as ML engineers building multi-turn tool-using or embodied agents who already understand GRPO-style policy optimization. Readers looking for an introductory treatment of reinforcement learning or agent architectures will find the methodology dense; the paper is not beginner-friendly.

Authors’ abstract

Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.