Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

AI summary

Overview

Research area: LLM-driven decision-making agents, memory-augmented reasoning, and non-parametric (parameter-free) optimization of agent behavior.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning formalism (MDPs, PPO, advantage estimation, trust regions), LLM agent architectures, and memory-augmented prompting methods.

Scope in one sentence: The paper proposes Skill-Pro, a framework that lets a frozen LLM agent autonomously grow a compact pool of reusable, natural-language procedural "Skills" from its own interaction trajectories, using a PPO-inspired but entirely non-parametric evolve-and-verify loop.

What This Paper Is About

LLM agents typically solve every task by reasoning from scratch at decision time, even when they have seen nearly identical situations before. Existing memory systems mostly act as "history books" — they store past trajectories, reflections, notes, or workflows that the agent must still re-read and re-interpret, which consumes context and reproduces the same instability and redundancy the memory was meant to fix.

Skill-Pro's goal is to convert passive episodic narratives into active, executable procedures that directly map recurring situations to action sequences, and to do so without updating any LLM weights, so that general-purpose capabilities are preserved.

Key Contributions

1. Procedural Skill formalization via a Skill-MDP. The authors extend the standard MDP into a Skill-augmented MDP (Skill-MDP) with a dynamic Skill pool Ω that serves as the agent's procedural memory, and define each Skill ω = ⟨ℐ_ω, π_ω, β_ω⟩ as a triple of activation condition, execution procedure, and termination condition expressed in natural language.

2. Non-Parametric PPO. A parameter-free optimization mechanism that replaces gradient descent on weights with three components: Semantic Gradients (natural-language refinement directions derived from trajectories via hindsight attribution), a PPO Gate (trust-region verification of candidate Skills using a clipped surrogate objective), and score-based pool maintenance (online scoring and pruning).

3. Reuse and performance gains with extreme memory compression. Across in-domain, cross-task, and cross-agent evaluations, Skill-Pro reports higher reuse rates and stronger task performance than five memory baselines and three reasoning baselines, while storing only 816 tokens.

4. Transparency artifacts. Visualized evolutionary lineages (skill lifespans, refinement events, pruning events, e.g., v₁ → v₁₃) and Skill distributions that show how procedural memory accumulates, refines, and is reused.

Code is available at https://github.com/Miracle1207/Skill-Pro.

Main Findings

• Reuse rates far exceed all baselines. In Table 1, Skill-Pro reports 0.925 ± 0.061 in-domain reuse, 0.825 ± 0.061 on Mastermind-v0-Hard, 0.900 ± 0.094 on Mastermind-v0-Extreme, 0.850 ± 0.094 with Gemma-3-4B, and 0.875 ± 0.112 with Qwen3-32B. The strongest baselines in those columns are RAG at 0.349, 0.441, 0.467, 0.111, and 0.146, and G-Memory at 0.091, 0.170, 0.092, 0.360, and 0.264.

• Memory footprint is dramatically smaller. Skill-Pro stores 816 total tokens with 102 ± 0 average tokens per unit (a Skill) and adds 273 ± 5 Δ prompt tokens per step. For comparison, G-Memory stores 40,510 tokens and AWM stores 391,706 tokens; Expel stores 294,447; A-MEM stores 200,129; RAG stores 116,527.

• Retrieval is less frequent, not constant. Skill-Pro's retrieval ratio is 0.591 ± 0.016, versus 1 ± 0 for RAG, Expel, and A-MEM, and 0.049 ± 0.009 for AWM and 0.097 ± 0.027 for G-Memory. The paper attributes this to Skills being temporally extended rather than per-step lookups.

• Performance gains hold under compression. In Table 2, Skill-Pro reaches 0.900 ± 0.105 on ALFWorld-Train and 0.909 ± 0.287 on ALFWorld-OOD, the highest reported in those columns (AWM follows at 0.700 and 0.900; G-Memory at 0.681 and 0.812). The paper explicitly highlights reaching 0.90 in ALFWorld.

• Mixed results on Mastermind difficulty tiers. On Mastermind-v0 Skill-Pro reports 0.606 ± 0.234, above G-Memory (0.577 ± 0.052) and ReAct (0.557 ± 0.059), but on Hard it reports 0.463 ± 0.210 against G-Memory's 0.406 ± 0.056, and on Extreme it reports 0.333 ± 0.118, below G-Memory's 0.356 ± 0.036.

• Cross-agent transfer works with different backbones. Skill-Pro reports 0.444 ± 0.161 with Gemma-3 4B-it, 0.615 ± 0.290 with Qwen3 32B, and 0.647 ± 0.236 with Llama-3.3 70B, the highest in each of those cross-agent columns among the listed algorithms.

• Skills and the evolution loop are both necessary. Removing Skills ("w/o Skill") drops performance from 0.606 to 0.388 (−36.0%). Removing Non-Parametric PPO ("w/o NP-PPO") drops reuse rate to 0.563 (−39.1%), performance to 0.482 (−20.5%), and online score to 0.0265 (−34.7%).

• Semantic Gradients drive candidate quality. Removing them ("w/o SG") cuts the PPO Gate Pass Rate by 30.2% (to 41.54%) and drops reuse rate to 0.306 (−66.9%), performance to 0.530 (−12.5%), and online score to 0.0015 (−96.3%).

• Without the PPO Gate the pool degrades. "w/o PPO Gate" admits 100.00% of candidates (a +68.1% change in pass rate) but yields the lowest reported performance among the NP-PPO ablations after FIFO, at 0.453 (−25.2%), with online score 0.0011 (−97.3%) and reuse rate 0.222 (−76.0%). The paper notes such unverified admission destabilizes training in the curves.

• Score-based pruning is the most critical maintenance rule. Replacing it with First-In-First-Out ("w/o Score (FIFO)") produces the most severe degradation: reuse rate 0.131 (−85.8%), performance 0.439 (−27.6%), and an online score of −0.0064 (−115.8%), which the paper reads as the pool replacing high-performing Skills with unproven newcomers.

• BFCL v4 results are referenced but not shown. The paper states that Berkeley Function Calling Leaderboard (BFCL v4) results are reported in Appendix C.2; those numbers are not present in the provided content.

Methodology in Plain English

The agent keeps a fixed-capacity pool of Skills. Each Skill is a short natural-language recipe with three parts: a cue saying when to use it, a list of steps saying what to do, and a condition saying when to stop and hand control back. At any moment, the agent either continues the current Skill or picks a new one from the pool, using either similarity between the current state and the Skill's cue, or a value-based choice among the top-k similar Skills. The underlying LLM is never retrained; it simply executes the selected Skill into primitive actions.

Learning happens by evolving the pool. After collecting a batch of trajectories, the system performs three operations:

1. Generate refinement directions. For each trajectory in which a Skill was used, the system looks back at what happened and attributes the outcome to one of the three Skill parts — the cue was wrong, the steps were wrong, or the stopping rule was wrong — producing a "Semantic Gradient," a natural-language suggestion for how to change that part. These per-trajectory suggestions are then consolidated across the batch into one averaged direction that keeps recurring patterns and discards conflicting one-off signals. Applying this direction yields a candidate Skill.

2. Verify before admitting. Because these candidates are LLM-generated from hindsight and may hallucinate, each candidate is scored with a PPO-style trust region test. The system computes an importance ratio between the candidate Skill and the Skill that was actually used, weights it by advantage estimates (return-to-go minus a running baseline), and clips the ratio into a band — exactly the PPO clipping idea, but applied to text-level Skills rather than network weights. Among N_c candidates, the best-scoring one is admitted only if its score is positive, meaning it is expected to beat the previous Skill while staying inside the trust region.

3. Prune by contribution. Each Skill accumulates an online score: the average advantage earned during the steps when it was active, divided by how often it was invoked (with at least 1). Skills with non-positive scores or that are duplicates are removed; if the pool still exceeds capacity, the lowest-scoring Skills are pruned. Since the baseline improves over time, this creates evolutionary pressure that retires obsolete Skills.

The authors evaluate on ALFWorld and TextArena's Mastermind-v0 (three difficulty tiers), with an additional BFCL v4 evaluation in an appendix, comparing against RAG, Expel, A-MEM, AWM, G-Memory, ReAct, CoT, and a State-based agent without external memory. Skills are learned with Gemma-2-9B on TextArena and reused across Gemma-3-4B, Qwen3-32B, and LLaMA-3.3-70B-Instruct; on ALFWorld everything uses Qwen3-32B. Results are averaged over 50 episodes per setting.

Why This Matters

The paper argues that the dominant memory paradigm for LLM agents — storing and retrieving episodic content — leaves the agent in the same inference-heavy loop it started in, no matter how large the memory grows. By shifting the stored unit from what happened to what to do, Skill-Pro targets both the compute cost and the reliability cost of long-horizon agents, and does so without the catastrophic-forgetting and over-specialization risks that the paper associates with parametric fine-tuning.

Real-world applications suggested by the paper's settings and framing:

• Tool- and API-calling agents, evaluated via the Berkeley Function Calling Leaderboard (BFCL v4), where repeated invocation patterns could be compiled into reusable procedures instead of being re-derived per call.
• Embodied and household assistants, matching the ALFWorld train/OOD environments, where recurring sub-tasks like navigation and object manipulation could be captured as compact Skills.
• Multi-turn strategic games and interactive tasks, matching TextArena's Mastermind-v0 tiers, where procedural shortcuts reduce per-step prompt load.
• Fleet or multi-backbone deployments, since cross-agent reuse is evaluated across Gemma-3-4B, Qwen3-32B, and LLaMA-3.3-70B-Instruct — implying Skills learned once could be transferred to cheaper or different models.

Industry relevance: The headline numbers are cost-oriented — 816 stored tokens versus 40,510 to 391,706 for baselines, and a retrieval ratio of 0.591 versus 1 for several baselines — which maps directly onto serving economics (prompt length, retrieval overhead, latency per step). The framework's compatibility with a frozen backbone also means it can be layered on top of deployed models without retraining pipelines or forgetting risks.

Future Directions

• Implicit procedural representations. The paper notes that its Skills are currently explicit and readable, similar to Claude Agent Skills, but can be hidden at the system level during execution, and states that directions toward implicit procedural representations are discussed in the appendix — leaving that transition as open work.

• Stronger Skill-selection policies. Selection by similarity and by value (top-k then highest Q) are described as simple instantiations of μ that could be replaced with RL-based retrieval policies; how much of the reported gain depends on the retrieval policy versus the Skill content is unresolved.

• Harder and more heterogeneous task regimes. On Mastermind-Extreme, Skill-Pro reports 0.333 ± 0.118, below G-Memory's 0.356 ± 0.036, and the reported variances on several Mastermind and cross-agent columns are large — indicating room to understand when procedural memory stops helping.

• Full generalization evidence. BFCL v4 is cited as an additional non-game, non-embodied evaluation, but its results are only referenced to Appendix C.2 and are not available in the provided content, so the breadth of generalization beyond text games and embodied environments remains to be confirmed from the full paper.

Target Audience

Researchers and engineers working on LLM agents, agent memory architectures, and non-parametric adaptation methods will get the most from this paper, particularly those already comfortable with reinforcement learning concepts such as MDPs, advantage estimation, importance ratios, and PPO clipping, since Skill-Pro's core mechanism is a direct analogy of PPO transposed into natural-language Skill editing. Practitioners building production agents with fixed, frozen backbones — especially those constrained by context length and per-step token costs — will also find the efficiency framing and the reuse metrics directly applicable, while readers looking for fully worked experimental appendix numbers (for example BFCL v4) will need the complete paper rather than the truncated content summarized here.

Authors’ abstract

LLM-driven agents excel at sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and instability. To bridge this gap, we propose Skill-Pro, a framework enabling agents to autonomously learn reusable procedural skills from interaction experiences without parameter updates. By formalizing a Skill-MDP, Skill-Pro transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, Skill-Pro sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that Skill-Pro achieves superior reuse rates and significant gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how Skill-Pro transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.