WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

AI summary

Overview

Research area: LLM agents, agent skill discovery and self-improvement, continual learning without parameter updates.

Technical level: Intermediate. The paper is readable for anyone familiar with LLM agents and prompting, though it includes a formal problem setup (datasets, trajectories, scoring functions) and notation for iterative skill evolution.

Scope: In one sentence: the paper introduces WikiSkill, a framework that co-evolves reusable agent skills with a persistent, non-rollback knowledge base ("wiki") compiled from execution traces, and evaluates it across five benchmarks and five models.

What This Paper Is About

AI agents can be given "skills" — reusable filesystem directories of instructions, scripts, and resources that encode domain procedures without changing model weights. Recent work automatically discovers such skills from the agent's own experience, but the insights guiding each skill update stay scattered across optimization histories, proposals, and rejection records, so they are hard to reuse systematically across iterations. WikiSkill's goal is to separate raw execution experience, accumulated knowledge, and executable skills into three layers, and to continuously compile experience into a persistent wiki that later skill updates can build on.

Key Contributions

1. A co-evolution framework. WikiSkill co-evolves agent skills with a persistent knowledge base that continually organizes and refines knowledge from agent experience. It organizes the workspace into a Raw Layer (immutable execution traces), a Wiki Layer (structured, compounding knowledge), and a Skill Layer (evolving procedural instructions).
2. Broad empirical validation. Across five benchmarks (LiveMathematicianBench, SealQA, SpreadsheetBench, OfficeQA, ALFWorld) and five models from the Qwen, Gemma, and Gemini families, WikiSkill consistently outperforms existing skill-evolution methods (Trace2Skill, EvoSkill, SkillOpt), with ablations confirming the importance of persistent knowledge accumulation.
3. A systematic study of skills versus model capability. The paper shows that skill evolution complements model scaling, that evolved skills transfer effectively across models and model families, and that skills evolved by another model can sometimes outperform self-evolved skills.
4. A framework-level distinction. The results separate two capabilities that self-evolution normally conflates: discovering useful procedural knowledge from experience, and effectively executing that knowledge at inference time.

Main Findings

• Consistent average gains over prior skill-evolution methods: WikiSkill achieves the highest average performance across all five models, improving over the strongest competing method by 3.3, 5.1, 10.0, 5.8, and 12.0 points for Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-4-31B, and Gemini-3.5-Flash, respectively. It improves over the no-skill baseline in most model-dataset pairs.

• Large domain-specific improvements: WikiSkill raises Gemini-3.5-Flash from 33.0% to 72.6% on LiveMath and from 50.5% to 76.6% on SpreadSheet, and raises Qwen-3.6-27B from 52.8% to 77.6% on ALFWorld.

• Baselines are less reliable: EvoSkill improves Qwen-3.5-9B on LiveMath (28.2% to 58.1%) but degrades Gemma-4-31B on the same benchmark (33.9% to 29.8%), and SkillOpt degrades Gemini-3.5-Flash on SealQA (29.4% to 28.2%).

• Skill evolution scales with model capability: Within the Qwen family, WikiSkill improves average performance by 12.3 points for Qwen-3.5-4B, 17.5 for Qwen-3.5-9B, and 23.9 for Qwen-3.6-27B. On SpreadSheet alone the three models improve by +6.5, +9.3, and +40.9 points.

• Skills can compensate for model scale: Qwen-3.5-9B with WikiSkill reaches 47.4% average accuracy, exceeding Qwen-3.6-27B without skills at 39.4%; Qwen-3.5-4B with WikiSkill reaches 38.5%.

• Dataset difficulty varies: For Qwen-3.6-27B, WikiSkill gains are 11.6 points on OfficeQA, 14.1 on SealQA, 24.8 on ALFWorld, 28.0 on LiveMath, and 40.9 on SpreadSheet. LiveMath gains range from 20.6 to 39.6 points across all five models; ALFWorld gains range from 14.0 to 29.3 points across the four models for which skills evolve. OfficeQA is hardest: larger models benefit (+11.6 for Qwen-3.6-27B, +12.1 for Gemini-3.5-Flash), while Qwen-3.5-4B slightly degrades because it struggles to execute multi-step long-context search workflows.

• Skills transfer across models and families, and can beat self-evolved skills: Qwen-3.6-27B skills raise Qwen-3.5-9B to 50.5% on SpreadSheet (versus 24.3% with no skills and 33.6% self-evolved) and raise Gemma-4-31B to 73.7% on LiveMath (versus 33.9% and 56.7%). On ALFWorld, Qwen-3.5-9B reaches 70.2% with a Qwen-3.6-27B-evolved skill versus 63.4% with its own. Transfer also occurs from smaller to larger models: Qwen-3.5-4B skills raise Gemma-4-31B to 73.1% on LiveMath and 66.9% on ALFWorld.

• Negative transfer is possible: Qwen-3.5-4B skills lower Gemini-3.5-Flash on SpreadSheet from 50.5% to 18.1%, while Qwen-3.6-27B skills raise it to 63.4%. The paper attributes this to low-level workarounds (e.g., single-line Python commands, string-conversion rules) that constrain stronger models, and to fragmented diagnostic procedures that exhaust the stronger model's interaction budget.

• A model can produce skills more useful to others than to itself: On OfficeQA, Qwen-3.5-4B skills decrease its own performance from 30.2% to 28.5% but improve Qwen-3.6-27B from 42.1% to 52.9%.

• Persistent knowledge is critical: In the Gemini-3.5-Flash ablation, disabling wiki access for the Inference Agent while giving the Skill Proposer wiki access raises average performance from 48.7% to 63.7% (+15.0%), including LiveMath from 51.3% to 72.6% and SpreadsheetBench from 49.9% to 76.6%. Removing persistent accumulation (no wiki for the Proposer, and no Wiki Maintainer) leaves the system at 48.7%.

• Wiki access during training hurts: Giving the Inference Agent wiki access during training rollouts reduces average performance from 63.7% to 60.9%, with LiveMath dropping from 72.6% to 64.8%. The authors hypothesize that the agent then draws task-solving knowledge from the wiki rather than the skills, making trajectories less informative for skill development.

• Wiki patterns accumulate while skills stay concise: Across models, Qwen models produce longer skills (118.9–128.6 lines) while Gemma-4-31B and Gemini-3.5-Flash produce more compact ones (45.1 and 81.2 lines). Wiki pattern accumulation ranges from 6.3–8.9 creations and 7.0–18.4 edits on average. Across benchmarks, SpreadSheet produces the longest skills (142.5 lines) and most wiki patterns (9.8); LiveMath produces the shortest skills (84.6 lines) and fewest patterns (4.4).

• Refinement continues over the whole run: Grouping accepted updates into early (iterations 0–1), middle (2–4), and late (5–7) stages, the initial stage accounts for 39%–52% of accepted updates across models and 39%–58% across benchmarks. On SealQA, 33% of accepted updates occur in the middle stage and 28% in the late stage.

• A special case: For Gemini-3.5-Flash on ALFWorld, all evolution methods match the no-skill baseline of 85.9% because that model reaches a 100% score on the validation split before skill evolution begins.

Methodology in Plain English

The agent workspace is split into three layers. The Raw Layer stores immutable execution traces — the agent's full step-by-step interactions, including reasoning, tool calls, tool outputs, and final answers. The Wiki Layer compiles those traces into structured knowledge: a patterns/ directory of markdown pages documenting failure modes and successful strategies, an index.md catalog, an evolution log (logs.md), and a skill-impact.md tracker recording each proposal's diff, validation score, and accept/reject outcome. The Skill Layer holds the active skills, where each skill directory contains SKILL.md (the full skill content) and PURPOSE.md (mapping the skill back to the wiki patterns that motivated it).

Each iteration runs four components. An Inference Agent executes training tasks using the current skills, with the full skill content injected directly into its system prompt and with wiki access deliberately blocked. A Wiki Maintainer then reviews a sampled subset of successful and failing traces alongside the existing wiki, performs root-cause analysis, extracts successful strategies, creates or patch-edits pattern pages, updates the catalog, and appends to the evolution log. A Skill Proposer — a multi-turn ReAct-style agent with a read_file tool — starts from the wiki index, the skill-impact tracker, and a summary of pass/fail outcomes, then reads specific pattern pages and raw traces on demand before issuing one atomic proposal: either a new skill or a patch-based edit to an existing skill. Finally, a Gating and Rollback mechanism applies the proposal, evaluates the candidate skill set on a validation split, and keeps it only if the validation score beats the best score so far; otherwise the skill set reverts. Crucially, the wiki is never rolled back, so accumulated knowledge persists even when skill updates are rejected. Evolution terminates early if validation reaches 1.0.

Evaluation uses five benchmarks — LiveMathematicianBench (mathematical reasoning), SealQA (web search), SpreadsheetBench (spreadsheet manipulation), OfficeQA (long-context document question answering), and ALFWorld (interactive embodied tasks) — and five models: Gemini-3.5-Flash, Qwen-3.5-4B/9B-Instruct, Qwen-3.6-27B, and Gemma-4-31B-It, with open-weight models deployed via vLLM. Baselines are Trace2Skill, EvoSkill, SkillOpt, and a no-skill condition. Every method starts from an empty skill set. The full evolution process is repeated three independent times per method, and reported scores are averages across the three resulting skill sets, with paired bootstrap testing at p < 0.05 (1,000 iterations).

Why This Matters

Research impact. The paper argues that accumulating agent experience into a persistent, structured knowledge representation — rather than leaving it scattered across optimization artifacts — is what makes iterative skill evolution effective, and it provides an ablation isolating that effect. It also reframes skill discovery and skill execution as distinct capabilities, which challenges the assumption that a model that evolves a skill is the best model to use it.

Real-world applications (potential, given the benchmark domains):

• Mathematical and quantitative reasoning assistants that accumulate reusable solution strategies over time.
• Web-search and research agents that learn better retrieval and verification procedures.
• Spreadsheet and document automation, where the paper shows the largest measured gains (SpreadSheet skills reach 142.5 lines on average; Gemini-3.5-Flash improves from 50.5% to 76.6%).
• Long-context document question answering over lengthy files, and interactive embodied or environment-控制 agents such as those tested on ALFWorld.

Industry relevance. Because skills are filesystem-based modules that require no parameter updates, the approach fits deployment settings where models are fixed, teams want auditable and reusable procedures, and knowledge should be shared across model releases. The transfer results suggest organizations can evolve a skill library once and reuse it across model scales and families, while the negative-transfer results caution that skills built around one model's weaknesses can hurt stronger models.

Future Directions

1. Reducing negative transfer. The paper shows that model-specific workarounds (e.g., single-line Python commands) can hurt stronger models. A natural next step is detecting or rewriting such workarounds into general procedures before transfer.
2. Understanding why wiki access during training degrades skills. The authors offer a hypothesis (knowledge leaks from the wiki instead of the skills, making traces less informative) but leave the mechanism unresolved.
3. Handling long-context skills for smaller models. On OfficeQA, Qwen-3.5-4B reverted to default reading behavior and slightly degraded. Making multi-step search workflows executable by smaller models, or matching skills to executor capability, is an open problem.
4. Broadening evaluation. The paper's dataset statistics and optimizer call-complexity comparisons live in appendices; extending the analysis to more domains, more models, and settings beyond the five benchmarks and five models studied would test generality.

Target Audience

Researchers and engineers working on LLM agents, agent skill libraries, prompt and procedure optimization, and continual learning without weight updates. It is also relevant to practitioners who maintain reusable agent procedures across multiple models or model versions, and to readers interested in how experience can be compiled into persistent, auditable knowledge rather than discarded after each iteration.

Authors’ abstract

Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.