UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents, specifically skill-augmented agents that reuse textual skills distilled from prior interactions.

Technical level: Advanced. The paper assumes familiarity with policy-gradient RL (GRPO, REINFORCE++), KL-regularized policy optimization, and LLM agent frameworks.

Scope: The paper introduces UniSkill, a shared-policy framework that jointly trains an actor and a skill proposer so that an agent's skillbank co-evolves with its policy, using a contrastive action-likelihood signal rather than extra environment rollouts to evaluate proposed skills.

What This Paper Is About

LLM agents can improve by saving reusable "skills" from past tasks in a skillbank and retrieving them later. When the actor (the agent that acts) keeps learning, however, it is hard to tell whether a newly proposed skill actually helps that actor, and directly testing each proposal requires expensive extra rollouts. UniSkill's goal is to learn skill proposals that are aligned with the current actor by reusing trajectories the actor has already collected, avoiding a fresh round of environment interaction per candidate skill.

Key Contributions

1. A shared-policy framework (UniSkill) in which one policy acts in the environment and proposes skillbank edits (Add, Update, or No Edit) from completed trajectories, plus skill-edit support regularization that keeps all available edit operations explored during joint training.
2. Contrastive action feedback, an actor-alignment signal that holds the current actor and recorded behavior fixed and measures how replacing the retrieved skill with a proposed skill changes the action log-likelihood gap between previously collected successful and failed trajectories from the same task, avoiding new rollouts for each proposal.
3. Strong empirical results on ALFWorld and WebShop among the compared methods, plus ALFWorld experiments showing UniSkill remains effective when the shared actor and proposer use a smaller backbone (Qwen2.5-3B-Instruct).
4. Analysis of the feedback signal, including an ablation isolating the R_align reward, a correlation study against measured rollout gains, and a study of support regularization's effect on skill-edit exploration.

Main Findings

• Final benchmark performance: UniSkill reaches 98.4% success on ALFWorld (mean ± std over three independent training runs: ±0.8) and a WebShop task score of 90.5 (±1.4) with 84.7% success (±0.5).
• Gains over RL-only baselines: UniSkill improves ALFWorld success over GRPO by 20.8 pp and over GiGPO by 7.6 pp.
• Gains over skill-augmented RL: A 12.0 pp improvement in WebShop success rate over SkillRL; 0.9 pp higher ALFWorld success than Skill1 with a comparable WebShop task score and 1.8 pp higher WebShop success.
• Comparison with the closest joint-training baseline: UniSkill exceeds Evolving-RL by 5.4 pp in ALFWorld success. On ALFWorld, UniSkill sustains stable joint training over 250 steps, whereas Evolving-RL makes strong early gains and then collapses (a risk also examined in Evolving-RL's original study). UniSkill's skillbank grows rapidly early and continues evolving through occasional additions and updates (non-overlapping 10-step means).
• Training efficiency: Timing measurements on ALFWorld compare per-step time for actor rollouts, proposal generation, skill evaluation, and skill critic calls. Rollout-based skill evaluation accounts for most of Evolving-RL's measured time, while UniSkill computes R_align from already-collected trajectories; the resulting savings outweigh UniSkill's longer actor rollouts and skill critic calls, giving lower total time across the reported stages.
• Ablation (Table 2, ALFWorld success): With retrieval at evaluation, UniSkill scores 98.4% versus 87.5% (Actor Only, frozen proposer), 89.1% (UniSkill without the R_align reward), and 32.8% (Skill Proposer Only, frozen actor). With retrieval disabled at evaluation, UniSkill retains 96.1% versus 84.4% for both Actor Only and the variant without the R_align reward, and 34.4% for Skill Proposer Only.
• Does R_align track actor alignment? Evaluated on critic-accepted Add and Update proposals with clipped R_align, using 32 rollouts per skill condition to estimate rollout gains: positive Spearman correlations between R_align and success-rate gains at training steps 25 and 75 (ρ = 0.312 and 0.367). Proposals with positive R_align have mean gains of 8.4 and 7.2 percentage points, versus −3.4 and −0.5 percentage points for negative-score proposals. However, 13/53 (24.5%) and 10/64 (15.6%) of positive-score proposals have negative observed gains at the two checkpoints, so R_align is informative in aggregate but not a guarantee per proposal.
• Support regularization sustains exploration: Without it, the skill-edit distribution collapses to a single operation (Add, in the reported run). With it, all three operations continue to be sampled with changing proportions, and validation curves show faster early improvement and higher later success rates (mean ± std over three independent runs per setting).
• Smaller backbone: With Qwen2.5-3B-Instruct as the shared actor and proposer, validation success trends upward, UniSkill-3B continues improving during later training, exceeds 90% success, and finishes above GRPO-7B.
• Out-of-distribution and critic robustness: UniSkill performs well on the ALFWorld out-of-distribution split with and without skill retrieval, with retrieval providing further gains (Appendix D.1), and achieves comparable WebShop performance with different skill critic models (Appendix D.2). The paper does not report numeric values for these two experiments in the main text.

Methodology in Plain English

The system uses one policy in two roles. As an actor, it plays an interactive task (household tasks in ALFWorld, shopping tasks in WebShop) conditioned on a skill retrieved from a skillbank. As a skill proposer, it looks at a finished trajectory together with an opposite-outcome trajectory from the same task and chooses one of three edits: leave the bank alone (No Edit), add a new skill, or revise the retrieved skill, along with the text of the skill content.

Training is joint. The actor is trained with GRPO using ordinary environment rewards. The proposer is trained with REINFORCE++ using a reward that combines three things: whether the proposal is well formed, whether a prompted skill critic judges the edit operation appropriate, and whether the proposed content is trajectory-grounded and reusable.

The distinctive part is how a proposed skill is scored for usefulness. Instead of running the actor again with the candidate skill, the method holds the pre-update policy, the recorded histories, and the recorded action tokens fixed, and only swaps which skill conditions the model. It then measures the average token-normalized log-likelihood of recorded actions. The actor-alignment reward R_align is the change in this quantity on a successful reference trajectory minus the change on a failed one, where both references are drawn from the same task and exclude the two trajectories shown to the proposer. A positive R_align means the candidate widens the likelihood gap between success and failure.

Because a single proposal-level advantage is applied to both the edit operation and the skill content, poor content can suppress an otherwise sensible operation. Skill-edit support regularization penalizes any available operation whose renormalized probability drops below a threshold p_min, discouraging premature collapse without forcing a uniform distribution. After the policy update, only edits that pass both critic checks and satisfy Δ+ > 0 and R_align > 0 are written into the skillbank; when multiple Update proposals target the same skill, the one with the highest R_align is applied.

Key settings: base model Qwen2.5-7B-Instruct for all compared methods, Qwen3-Embedding-0.6B for retrieval, DeepSeek-V4-Pro as the skill critic, η = 0.05 and p_min = 0.1, starting from an empty skillbank, and evaluating under ALFWorld's official valid_seen split with a common protocol.

Why This Matters

Impact on research. The paper targets a credit-assignment problem specific to co-evolving policies and memory: as the actor changes, past signals about a skill's value become stale, and rerunning the environment for every candidate is expensive. Contrastive action feedback offers a cheap, actor-specific proxy, and the analysis honestly documents both its aggregate usefulness and its per-proposal failure rate. The support-regularization idea—preventing a shared training signal from starving exploration of discrete edit operations—is transferable to other learned memory-management and tool-use settings.

Real-world applications (illustrative, matching the paper's benchmark domains and framing):

• E-commerce shopping assistants that must search, inspect, and select products from natural-language requests.
• Embodied or simulated household assistants that navigate and manipulate objects across many task types.
• Enterprise agents that accumulate and maintain a shared library of reusable procedures rather than re-deriving them per task.
• Assistants in settings where calling the environment for extra evaluation is costly or slow, so offline reuse of already-collected trajectories is preferable.

Industry relevance. The method is a training recipe, not just a prompting trick: it jointly optimizes an actor and a skill proposer and removes an expensive evaluation step from the training loop. The reported efficiency comparison against Evolving-RL, and the finding that a 3B backbone remains effective, point toward more affordable deployment and iteration at smaller model sizes. The authors report that the implementation is available at a public repository.

Future Directions

• Improving per-proposal reliability. Since 24.5% and 15.6% of positive-R_align proposals at the two checkpoints had negative observed gains, a more reliable alignment proxy or a complementary check is an open problem.
• Generalizing beyond the two benchmarks. The paper evaluates on ALFWorld and WebShop; extending actor-aligned skill proposal learning to other interactive domains, and to the ALFWorld out-of-distribution split in more depth, remains to be shown.
• Reducing dependence on a prompted skill critic. UniSkill relies on an external critic (DeepSeek-V4-Pro) for operation appropriateness and content support; whether learned critics can replace it at comparable quality is not established.
• Tuning the exploration machinery. The thresholds η = 0.05 and p_min = 0.1 and the regularization weight λ_sup are fixed; how sensitive results are to these settings, and whether they should adapt as the skillbank matures, is left open.

Target Audience

Researchers and engineers working on LLM agents, reinforcement learning for language models, and lifelong or memory-augmented learning systems. Practitioners building skill- or experience-reuse pipelines for multi-turn interactive agents will benefit most from the training recipe and the ablation evidence, while readers interested in credit assignment for co-evolving policies and memory will find the contrastive action feedback formulation and its empirical analysis the central contribution. Beginners can follow the high-level motivation and results, but the methodology section requires prior familiarity with policy-gradient objectives and KL regularization.

Authors’ abstract

Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone. Our implementation is available at https://github.com/LimOkii/UniSKill.