Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement

AI summary

Overview

Research area: LLM agent tooling — specifically the curation and lifecycle management of reusable "skill" libraries (documents, scripts, and references that agents retrieve at inference time), with a focus on measurement validity and reliability.

Technical level: Advanced. The core idea is intuitive, but the paper leans on psychometric theory (classical test theory, generalizability theory, mastery testing, Cohen's κ), formal propositions about risk, and a variance-decomposition argument. Readers without background in measurement theory will still follow the argument, but the proofs and reliability decomposition require care.

Scope: The paper argues that retiring an agent skill should be decided by a model's unaided capability score rather than by the skill's measured gain, and it supports this with a formal impossibility result, a 1,251-skill dataset scored under four models, reliability analysis, and an execution test on SkillsBench.

What This Paper Is About

Public hubs of agent skills now hold on the order of 10^5 entries and serve many different models. As models improve, a hub needs to know which skills a given model has already absorbed — meaning it can perform the capability without the skill — so that entry can be retired for that model while remaining available to others. Existing published rules decide this from the gain (aided score minus unaided score), but the paper shows the gain is near zero both when a skill has been internalized and when neither the model nor the skill supplies the capability at all — a state the authors call uncovered — so gain-based retirement cannot distinguish "absorbed" from "broken."

Key Contributions

1. A diagnosis of why gain-based retirement fails. The authors formulate retirement as a per-model certificate of unaided capability and prove that any criterion which vanishes when the two arms are equal — including leave-one-out and Shapley attributions on a single-skill task, plus the difference, log-ratio, and mutual information — gives one verdict across the whole diagonal where aided and unaided scores coincide, and therefore cannot separate uncovered skills from internalized ones. A second proposition shows that adding more runs removes noise within a task but not this error.

2. A measurement protocol and released dataset. The protocol withholds the skill, scores the model on an executable task synthesized from that skill (by a generator from a different model family, so the content is not trivially shared), applies a validity gate that admits a task only when another model gains from the skill, and adds extra tasks for skills near the decision threshold. The release covers 1,251 skills with tasks, rubrics, and item-level scores under four models.

3. An audit of published rules on real data. The paper reproduces auditable published criteria on static data and measures what each removes, both on its own adversarial library and on Tessl-single, and repeats the audit with each of the four models as the target in turn.

4. Reliability and validity analysis of the label. A three-facet reliability decomposition (items, runs, tasks), a measurement-budget proposition, a judge-dependence check with two independent judges/graders, and a correction for the difficulty gate that dropped skills before scoring.

Main Findings

• The gain is blind to the uncovered state. The gain is near zero both when a model has absorbed a skill and when neither the model nor the skill delivers the capability. On the gated adversarial library the 18 skills at unaided/aided (0, 0) and the 82 at (1, 1) have the same gain, and a band rule retires both groups.

• The uncovered state is real and sizable. It holds 8.6% of the gated adversarial library and 5.9% of Tessl-single, with mean scores of 0.10 and 0.15 unaided and 0.18 and 0.27 aided. The band |A − U| < 0.10 holds 27.8% and 15.0% of the two libraries, and within it the unaided score runs the full range from 0 to 1 (standard deviation 0.331 and 0.277).

• The published band rule removes over half the uncovered skills. With Qwen3.6-27B as target and one task per skill, the band |A − U| < 0.10 retires 17.9% of the uncovered state on the adversarial library — 57.5% of the uncovered state, with a skill-level bootstrap interval of [46.2, 68.7] — and 47.6% of the uncovered state on Tessl-single. The uncovered state is 2.1 times as dense inside the band as in the library at large.

• The effect holds for every model as target. On 844, 834, 839, and 836 gated skills, the band removes 57.5%, 62.8%, 65.4%, and 58.9% of the uncovered state on capability scores, and 79.5%, 98.7%, 91.7%, and 86.9% on the binary outcomes its source rule uses.

• The level rule costs little and buys more. The level rule (retire when U ≥ 0.8) cannot retire an uncovered skill by construction, and it retires 15.9 points more of the internalized state than the band, with a paired interval of [10.9, 20.7]. Across the four targets the level rule removes 13 to 16 points more internalized skills than the band.

• The result survives separating measurement from state. Reading the verdict on one task and the state on the other, over the 124 skills scored on two tasks, the band retires 45% of the 29 decisions on skills uncovered on the other task and the level rule 14% — a paired difference of −31 points [−52, −7].

• A validity gate is necessary. The gate admits 844 of 1,165 scored tasks (72.4%), but only 52.6% of those with U ≥ 0.8 — so nearly half the apparent internalization at the top of the range comes from tasks no other model benefits from. A text-only predictor separates high from low unaided scores at 0.798 AUC ungated and 0.656 gated, indicating that in an ungated library text mostly tracks whether the task is trivial.

• Reliability lives in tasks, not runs or items. Internal consistency over items is 0.907, test–retest over runs is 0.90 at the three-run mean, but parallel-forms reliability over a second task is only 0.41. Variance components are 0.046 (skill), 0.053 (task), 0.036 (run) — 34%, 39%, 27% of a single run's variance. Three tasks with one run each misclassify 11.4% of the gated library; one task run three times misclassifies 16.1%; tripling runs on a single task lowers it only to 15.2%. The optimal allocation is n_r* = 0.83·sqrt(c_t/c_r) runs per task.

• Targeted measurement is cheaper. Adding tasks only within 0.2 of the threshold (41% of the gated library) reaches 11.1% misclassification at three quarters the cost of tripling tasks everywhere.

• The level verdict reproduces better than the band. Across tasks for the 124 doubly-scored skills, Cohen's κ is 0.42 for the level rule against 0.23 for the band, with a paired bootstrap interval of [0.01, 0.38] for the difference. The gain itself has reliability 0.15 across tasks, versus 0.85 for the aided score and 0.80 for the gain across runs of one task; the two arms share 87% of what persists across tasks and the gain keeps 0.011.

• Verdicts are model-specific. Between pairs of models, verdicts flip for 14% to 23% of skills, and 37% of skills receive contradictory verdicts somewhere among the four. Resampling a model's own three unaided runs flips its verdict for only 5% to 9% of skills. With one task per skill, the skill together with its task explains 62% of the variance in the unaided score, the skill×model interaction 15%, and overall model strength 0.4% — so per-model threshold tuning would not remove the disagreement.

• On SkillsBench, retirements rest on probes nobody passed. The benchmark analysis builds one internalization probe per each of 200 shipped skills (182 scorable), fixes analysis to the 92 probes where no rubric item is given away (59 admitted by the gate), and finds 28 of the band's 31 retirements rest on probes that no run passed in full, while retiring on the unaided score costs no detectable reward.

• Retirement does not damage retrieval. With a dense retriever and reranker ranking each task's own skill among 10,000 corpus distractors (credited when in the top three), the level rule retires 19% to 29% of the library across three models and retains 91% to 97% of that utility, against 70% to 80% for a random retirement of the same size.

• Judges are largely consistent but not exact. Re-scoring 444 unaided trajectories with DeepSeek-V4-Flash gives a 0.89 correlation with the target model's judgments, a 0.009 mean difference, and 91.4% verdict agreement, with disagreements split 20 to 18. A blind regrade by Claude Opus 5.5 of 99 items near the threshold found 9% of decidable full marks unsupported (15% counting partly met criteria) and 2 of 20 zero marks wrong; such unsupported full marks lift U near τ by about 0.1.

• Item validity cannot be fully established from text. The text adjudicator reaches 0.85 precision and 0.81 recall against a 120-item author annotation, but the counterfactual condition ("would the model pass without this information?") is not identifiable from skill, task, and item. Among the 6,802 capability items of gated tasks, 10.0% are satisfied without the skill by all four models on every run, concentrated in 29.4% of skills at a median 30% of an affected skill's items, and no skill has all items in this class.

• The difficulty gate biases the library, and the correction is modest. The gate removed 859 of 3,000 attempted skills; regenerating tasks without it for 115 of them puts 69.6% above the threshold unaided, versus 40.4% of the 1,323 retained skills. Post-stratifying lowers the uncovered base rate from 8.6% to 7.0% and its share of band retirements from 17.9% to 9.9%, while the share of uncovered skills the band removes barely moves, to 57.2% [45.7, 68.3].

• Grading all items instead of capability items does not change the conclusion. Scoring all task-completion items gives an ordering correlating 0.93 with the capability-item ordering, and the band still retires 51.7% of the uncovered state while the level rule still retires 15.7 points more of the internalized state.

• Re-labelling for a new model is cheaper. A new deployment model reuses the tasks, item selection, and witness evidence, paying only for its own unaided arm — 61% less than labelling from scratch.

Methodology in Plain English

The researchers start from a simple two-arm comparison: run the agent on a task with the skill and without it. That gives an aided score (A) and an unaided score (U). The gain is A − U. Sorting skills by these two numbers yields three states: the skill works (aided high, unaided low), the skill is internalized (both high), and the skill is uncovered (both low). The gain is near zero in the first and third cases, which is the root of the problem.

To measure the unaided level properly, they take each skill and have a separate generator model — deliberately from a different model family — write one executable task with hidden rubrics. The task states the goal but withholds the convention or answer the skill teaches, so a model lacking the skill must supply that content itself. A difficulty gate regenerates tasks the target already passes on more than half their rubric, and a critic rejects tasks that cannot be performed as written. Only rubric items that genuinely require information stated in the skill are scored ("capability items," selected by an adjudicator model).

A validity gate then checks whether the task actually tests the skill: the task is admitted only if at least one witness model gains from the skill on the same items by more than 0.1. This distinguishes "the model has the capability" from "the task is easy." Witness evidence accumulates and does not expire.

Four open-weight models are used: Qwen3.6-27B as target, with Qwen3.8-27B, Qwen3.5-397B, and DeepSeek-V4-Flash as witnesses and as alternative deployment models. The unaided arm is run three times per model, the aided arm three times for the target and twice for witnesses, in a tool-use loop of at most 40 turns, scored by a single judge (Qwen3.6-27B) against the rubrics. 135 skills carry a second task from a later batch, used to check whether verdicts reproduce across tasks.

The thresholds used throughout are 0.7 for "high," 0.4 for "low," τ = 0.8 for the retirement threshold, γ = 0.1 for the gate, and δ = 0.10 for the band width. Reliability is decomposed into item-level consistency, run-to-run stability, and task-to-task stability, and a proposition converts this into a measurement budget showing how many tasks versus runs to buy. The final procedure (Algorithm 1) measures U, adds two tasks if U is within 0.2 of τ, consults witnesses only before retiring, and runs the aided arm only to flag uncovered skills for repair rather than to decide retirement.

Why This Matters

Impact on research. The paper reframes skill retirement from a ranking problem to a mastery problem, and shows formally that a whole family of attribution-style criteria cannot solve it no matter how many runs are collected. That is a structural result, not a tuning issue. It also imports measurement theory (reliability coefficients, generalization theory, mastery testing) into agent evaluation, where the norm has been to report a single benchmark number with no reliability coefficient — the authors note that none of the criteria they survey reports one.

Real-world applications:

• Skill hub and registry curation. Operators of large public skill libraries who must decide, per model version, which entries to deprecate, which to keep, and which to flag as broken.
• Model upgrade migrations. When a provider ships a new model, the hub can re-run only the unaided arm against existing tasks and witness evidence, at 61% lower cost than labelling from scratch.
• Skill authoring and repair triage. The uncovered state doubles as a repair flag: a skill that fails with and without itself is a candidate for fixing, not deletion.
• Retrieval and context-budget management. Retiring genuinely redundant skills frees retrieval slots and context without a measurable drop in task success, provided retirement is decided on the unaided level rather than the gain.

Industry relevance. The finding that 37% of skills receive contradictory verdicts across four models means a hub cannot maintain one global "deprecated" label — retirement must be per-model, and a stronger model's verdict cannot stand in for a weaker one's. The measurement-budget result also has direct cost implications: at the observed cost ratio, one run per task is optimal unless generating a task costs more than about three runs, so spending on more runs of the same task is largely wasted.

Future Directions

• Cheaper task synthesis at scale. Parallel-forms reliability of 0.41 across tasks, against 0.907 across items and 0.90 across runs, means the binding constraint is task generation. The paper shows same-batch tasks agree more (0.549) than cross-batch pairs (0.285), so how to generate diverse-but-comparable task batches remains open.

• Resolving the counterfactual item problem. The third condition for a capability item — that the model would fail without the skill's information — is shown to be unidentifiable from skill, task, and item. The 10.0% of capability items that all four models satisfy unaided remain unremovable by text analysis, and a better behavioural or interventional test is an open measurement problem.

• Judge and rubric robustness near the threshold. Unsupported full marks lifted U by about 0.1 near τ, which is exactly the margin between τ and the "high" threshold h. Tighter sandbox-checked rubrics, or ensembling judges, would directly shrink misclassification near the decision boundary.

• Policy design beyond a hard threshold. The authors recommend a high threshold, extra measurement near it, and a reversible soft retirement, because U averages over items and a skill can be partly internalized. How soft retirement, repair flags, and per-model re-reading should interact in a live hub is proposed as a procedure but not evaluated downstream — and the truncated text does not report the outcome of the SkillsBench execution test in full, only that retiring on the unaided score costs no detectable reward.

Target Audience

Researchers and engineers working on LLM agent infrastructure — skill libraries, tool registries, context and retrieval management — and evaluation scientists who care about whether a benchmark-derived label is valid and reproducible before a system acts on it. It is also relevant to anyone applying psychometric methods (reliability decomposition, mastery testing, attenuation corrections) outside education, and to maintainers of public skill hubs who need an operational, per-model retirement policy rather than a single leaderboard score.

Authors’ abstract

LLM agents retrieve skills from public hubs that hold up to about 100,000 entries and serve many models. As models improve, a hub must find the skills a given model has internalized, meaning it can do without them, and retire them for that model, yet internalization has not been measured with known reliability. Published rules that compare runs with and without a skill retire on the gain alone. The gain is near zero both for an internalized skill and when neither the model nor the skill delivers the capability, a state we call uncovered. We instead withhold the skill, score the model on a task synthesized from it, and trust a high unaided score only when another model gains from the skill, since nearly half of high scores come from tasks no model benefits from. Under four models, a published gain rule applied with one task per skill removes over half of the uncovered skills. On SkillsBench, 28 of its 31 retirements rest on probes no run passed in full, while retiring on the unaided score costs no detectable reward. Unaided verdicts reproduce across tasks more often than the gain rule's (Cohen's kappa 0.42 against 0.23) but differ across models, so a hub must read them per model. We release 1,251 skills with tasks and item scores under four models.