Evolving to Match Capabilities: Improving Harness Self-Evolution via Skill Adaptation

AI summary

Overview

Research area: Machine learning / LLM agents — specifically agent harnesses, reusable "skills" injected into agent context, and harness self-evolution (agents rewriting their own scaffolding from execution feedback without changing model weights).

Technical level: Advanced. The paper assumes familiarity with LLM agent harnesses (e.g., OpenHands, Claude Code), skill files such as SKILL.md, benchmark verifiers and LLM-as-judge scoring.

Scope: An empirical study showing that a fixed skill's benefit depends on the executor's capability, plus a "meta-skill" rewriting protocol that adapts skills to weaker executors, evaluated both as one-shot rewriting by a strong external model and inside a five-round self-evolution loop across seven models and three benchmarks.

What This Paper Is About

Agent harnesses improve by distilling reusable skills from execution experience, but the authors find that a skill which helps a strong model can leave a weak model unchanged or worse. Their diagnosis is that the guidance is often already present in the skill — it is the executor's ability to act on it that differs. The goal is to make harness evolution capability-aware by reformulating existing skills so that weaker executors can actually follow them.

Key Contributions

1. A quantified capability gap in skill benefit. Across seven models and three independently constructed benchmarks (SkillsBench, DocOps, Tessl), the authors identify 103 task–skill pairs (out of 607 total tasks: 87 SkillsBench, 186 DocOps, 334 Tessl) where skills improve frontier models by 29.2 points on average but weak models by only 1.2 points; on individual models, skills improve GPT-5.6-luna by 31.1 points while degrading Gemma-4-E4B by 8.8 points.

2. A failure-mode taxonomy for weak executors. Reading the trajectories of three weak models on the 103 pairs, they assign 131 labels over four recurring failure modes — tool execution, context management, instruction following, and verification — and report that in 86% of failed weak-model runs the violated requirement is already stated in the injected SKILL.md.

3. A fixed, task-independent meta-skill. The four failure modes are converted into concrete rewriting instructions (e.g., separate callable tools from documents; state task, allowed scope and delivery path up front; give each requirement an action, target and completion condition; end with an independent verification loop). The meta-skill was developed with Claude-Opus-5 as a writing assistant.

4. Two validated uses of the meta-skill. One-shot skill adaptation by a capable external rewriter with no execution evidence, and integration into a five-round self-evolution loop where each model revises its own skill from its trajectory and verifier feedback.

Main Findings

• Skill benefit tracks base capability. Averaged over all 607 tasks, skills yield a small gain for every model with no clear tier pattern; on the 103 skill-sensitive pairs, the skill lifts frontier and moderate models by up to 31.1 points yet leaves weak models nearly unchanged and hurts the weakest one. Figure 1 reports +27 points on average for strong- and mid-tier models versus +1 point for the weak tier on the 103 pairs.

• The gap is not missing information. 86% of failed weak-model runs violate a requirement already written in the injected skill, so harness evolution must consider whether a skill's formulation matches the executor, not only whether it captures useful experience.

• One-shot adaptation works only with a capable rewriter. With the meta-skill, Kimi-K3 raises the average score over the three weak executors from 0.294 to 0.387 (Qwen3.6-flash 0.529→0.616, Gemma-4-E4B 0.029→0.170, Qwen3.5-9B 0.323→0.374); GLM-5.3 achieves 0.379. Weak models rewriting their own skills one-shot fall below the original (average 0.271).

• The meta-skill adds value beyond generic rewriting. Against a one-sentence rewriting instruction with the same rewriter (Kimi-K3), the meta-skill adds 5.1–6.0 points on average over the three weak executors and accounts for more than half of the average gain; the generic instruction improves only two of three executors and leaves Qwen3.5-9B slightly below the original skill.

• Self-evolution benefits from the meta-skill across all seven executors. After five rounds, every executor from Gemma-4-E4B to Kimi-K2.5 scores 2.0–5.1 points higher with the meta-skill than in the same loop without it. Round-5 gaps for larger executors: +0.046 on Qwen3.5-27B, +0.020 on Gemma-4-26B, +0.025 on Qwen3.5-plus, +0.035 on Kimi-K2.5. Without the meta-skill, Qwen3.5-9B peaks at round 3 and falls back almost to baseline, and Gemma-4-E4B stays below its baseline every round.

• Different settings repair different failure modes. One-shot skill adaptation gives mean gains of +0.056 (Execution, n=51), +0.055 (Instruction, n=35) and +0.103 (Context, n=28), with only +0.003 on Verification (n=17). In self-evolution, Verification gains average +0.104 — 0.101 larger than in one-shot adaptation — because trajectories reveal whether a check was actually performed.

• Remaining failures belong to the executor. Revised skills can state the correct procedure while the model still edits a script without rerunning it, repeats a failed call, or drops constraints; in the paper's worked example, Qwen3.6-flash passes four of five rounds with the meta-skill but in the failing round fixes the script and never reruns it.

Methodology in Plain English

The researchers first ran seven models on skill-augmented tasks with and without the injected human-written skill, so they could measure each skill's "gain" per model. They grouped the models into weak, moderate and frontier tiers by their no-skill performance, then kept only the task–skill pairs where a skill clearly helps the stronger tiers more than the weak tier (a threshold of 0.10 on both the tier gain and the tier gap, dropping pairs where the weak tier already scored ≥0.80 with the skill). That filter produced the 103-pair testbed.

Next they read weak models' failed trajectories and tagged the first observable failure with one of four labels, which became a checklist for rewriting. From that checklist they wrote the meta-skill: a fixed system prompt telling a rewriter how to restructure a skill. In the one-shot setting, a strong external model received the task, the original skill and the executor's tool list — no trajectories — and returned either a full set of revised files or NO_CHANGE. Returned files were screened for loadability (valid path, unchanged skill name in the YAML frontmatter, under a 64,000-character budget) and for answer leakage (specific test-only tokens), falling back to the original skill if a check failed.

In the self-evolution setting, the executor rewrote its own skill for five rounds, each round seeing the previous round's score, trajectory digest and verifier output, with gains measured against a same-time-window baseline that reruns the original skill. Three benchmarks with disjoint construction and scoring protocols (deterministic tests, document-state checks, LLM judge with rubric) were used so a conclusion is reported only if it replicates.

Why This Matters

Impact on research. The paper reframes harness benefit as a joint property of the skill and the executor, and argues that harness evolution should be evaluated per capability tier rather than with one representative model. It also shows that failure evidence changes what weak self-rewriters can do: the same meta-skill that hurts weak models in one-shot rewriting helps them once trajectories are available.

Real-world applications:

• Deploying one vendor- or expert-written skill set across a mixed fleet of models, where small/cheap models run the same skill as frontier models.
• Document and office-automation agents (the DocOps setting: xlsx, docx, pptx, pdf edits) where delivery paths and output formats are frequently missed.
• Developer-tooling agents for vendor APIs (the Tessl setting, e.g., Stripe, Vercel, ClickHouse, Sentry, Shopify, Firecrawl) where rubric-graded outputs have no single correct artefact.
• Self-improving internal agent platforms that must revise their own prompts and procedures without retraining or fine-tuning.

Industry relevance. Skill registries already distribute skills written once and shipped unchanged; this work gives a concrete, cheap procedure (a rewriting prompt plus validity gates) for adapting them downward in capability, and a diagnostic checklist that practitioners can reuse.

Future Directions

• Tracking whether each revision actually resolves the failure that motivated it, alongside task score, so later updates target persistent problems and useful revisions are retained.
• Extending failure-mode-guided adaptation beyond skills to tool interfaces and agent workflows.
• Testing adaptation under skill retrieval and progressive disclosure, which this work explicitly leaves out of scope (skills are evaluated as already present in context, and multi-skill tasks are treated as bundles).
• Determining how much of the 0.02–0.05 round-5 gap is signal: each round is run once on all 103 pairs, so the authors rely on the sign being consistent across all seven executors rather than on the magnitude, and the fixed five-round budget was not tuned.

Target Audience

Researchers and engineers working on LLM agent harnesses, agent skills, and self-improving agent systems; benchmark designers interested in capability-stratified evaluation; and practitioners who maintain shared skill libraries deployed across models of different sizes. Readers without background in agent harnesses or benchmark verification will find the framing accessible but the experimental setup dense.

Authors’ abstract

Harness self-evolution offers a practical route for LLM agents to improve from execution experience without updating model parameters. Its effectiveness depends on whether models can successfully act on the guidance provided by their harnesses. However, we find that harness benefits are not uniform across model capabilities: skills that improve stronger models can provide little benefit or even degrade performance for weaker ones. Across seven models and three benchmarks, we identify 103 task-skill pairs where skills improve GPT-5.6-luna by 31.1 points but degrade Gemma-4-E4B by 8.8 points. In 86% of failed weak-model runs, the violated requirement is already stated in the skill. This motivates skill adaptation: reformulating existing skills to match the executor's capabilities and make their guidance easier to execute. To achieve this, we diagnose failures in tool execution, context management, instruction following, and verification, and distill these diagnoses into a fixed meta-skill that guides adaptation during harness self-evolution. Guided by the meta-skill, a capable external rewriter, Kimi-K3, adapts skills without execution evidence, improving the three weak executors by 5.1-14.1 points over the original skills. We then incorporate this adaptation into a five-round self-evolution loop, where each model revises its own skill from execution trajectories and verifier feedback. Across seven executors, from Gemma-4-E4B to Kimi-K2.5, the meta-skill consistently improves final scores by 2.0-5.1 points over the same loop without the meta-skill. Our findings provide practical design guidance for developing more effective harness self-evolution methods.