Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills

AI summary

Overview

Research area: LLM agents — specifically how agents acquire reusable, transferable "Agent Skills" (packaged instructions and executable resources) without updating model weights.

Technical level: Advanced. The paper assumes familiarity with LLM agent harnesses, trajectory distillation, and benchmark design.

Scope: The paper introduces Socratic Skill Learning (SSL), a protocol where an Educator model guides a Student model during task practice, and SkillTransferBench, a 35-family / 408-case benchmark for testing whether learned Skills transfer to held-out tasks.

What This Paper Is About

Agent Skills let LLM agents reuse task knowledge across sessions, but acquiring a transferable Skill requires evidence about not just how to do a procedure but when it applies — evidence a single successful attempt often leaves untested. The authors ask how external expert guidance should be timed and organized while a learner is still gathering that evidence, rather than after it has already committed an untested assumption to a reusable rule. Their answer is to interleave expert teaching with Student practice, keeping the Student as the sole environment operator and the sole author of the Skill package.

Key Contributions

1. SkillTransferBench: a benchmark of 35 task families and 408 executable cases across six domains (Systems, Services, Data Processing, Scientific Computing, Finance, Office Automation), where each family pairs an anchor task with a public specification of invariants and permitted variation, and evaluation uses fresh solver sessions on held-out family members. The main cohort is 22 families / 155 cases; a lite subset is 10 families / 77 cases.

2. Socratic Skill Learning (SSL): an acquisition protocol in which an Educator with read-only access to recent Student work and the live Skill can explain, challenge, diagnose, propose checks, or stay silent, while the Student questions, tests, adapts, or rejects the advice and remains the only agent that operates the environment and writes the package.

3. Two instantiations of SSL: Fixed-Window SSL, which reviews every g completed Student tool calls, and Issue-Guided SSL, which maintains a persistent "Learning-Issue Tree" of unresolved questions and requests review only when an issue opens, changes, or reopens.

4. Cross-model evaluation plus process analysis and a case study, covering three model pairs (Qwen, DeepSeek, Kimi) and linking Educator lessons to Student writing and package revision, and review scheduling to supervision cost.

Main Findings

• SSL leads on the Qwen main cohort (22 families, 155 cases). Fixed-Window SSL reaches 0.801 utility and Issue-Guided SSL reaches 0.805, versus 0.723 without a Skill, 0.761 for adapted Trace2Skill, and 0.751 for adapted SKILL-KD. The two SSL schedules rank first and second overall and in each of the six domains.

• The margin over distilled baselines is measured with uncertainty. Fixed-Window SSL exceeds adapted Trace2Skill by 3.99 percentage points (95% paired family-bootstrap CI [1.45, 6.49]) and adapted SKILL-KD by 4.99 points.

• Domain-level gains vary. On the All column (0–100 scale), Fixed-Window SSL scores 80.1 versus a No Skill score of 72.3 (+7.8). The largest domain gain is Scientific Computing (+10.2) and the smallest is Office (+3.7). Issue-Guided SSL's largest gain is Services (+12.8) and smallest is Office (+4.2).

• Improvements carry through to complete solutions. Family-macro full-pass rate, which requires every private check to pass, rises from 32.50% with No Skill to 49.49% with Fixed-Window SSL and 50.82% with Issue-Guided SSL.

• Guidance during practice beats guidance after the fact. On the shared lite subset (10 families, 77 cases), Fixed-Window SSL outperforms Post-hoc Teaching on all three model pairs, and both interleaved protocols surpass direct Expert Authoring. Post-hoc Teaching trails Expert Authoring in every model pair. On the main cohort, interleaving's advantage over Post-hoc Teaching is 5.31 percentage points with 21% fewer acquisition tokens.

• Cross-model gain over No Skill on the lite subset: Fixed-Window SSL +4.1 (Qwen), +3.1 (DeepSeek), +8.2 (Kimi); Issue-Guided SSL +4.5, +2.5, +7.4; Expert Authoring +1.1, +1.2, +5.8; Post-hoc Teaching −0.7, +0.5, +4.1.

• Teaching demand falls as competence grows. Across 630 reviews in 66 Qwen Fixed-Window SSL runs with 3 rounds each, the fraction of regular reviews delivering a lesson falls from 73.0% in round one to 16.1% in round three, and advice shifts from reasoning toward diagnosis. Packages still change in 98 of 99 rounds following a perfect anchor score.

• Lessons are followed by measurable Student behavior. Compared with silence, lessons precede a 23.9-point higher probability of file writing or editing on the next call and a 38.6-point higher probability of a Skill change over the next ten calls, after adjusting for round.

• Fixed schedules get expensive. Average tokens per review grow 5.3-fold from the first to the last round as context accumulates, and reviews returning silence account for 85.8% of final-round review tokens before follow-up dialogue.

• Issue-guided review preserves utility at lower supervision cost. Issue-Guided SSL reaches 0.805 utility versus 0.801 for the standard g = 10 window, using 61.7% fewer Educator tokens and 16.9% fewer total acquisition tokens, achieving 2.75 times the transfer gain per Educator token. Compared with g = 10, it reduces review count by 47.8% and tokens per review (including follow-up dialogue) by 26.7%.

• Simply changing the interval does not substitute for issue guidance. g = 5 increases Educator cost and g = 20 reduces it, but both yield lower utility than Issue-Guided SSL.

• Cross-model savings hold. On DeepSeek and Kimi, Issue-Guided SSL uses 63.8% and 77.4% fewer Educator tokens respectively while retaining utility within one percentage point of Fixed-Window SSL. Total acquisition tokens also fall for Kimi, but additional Student work raises the total for DeepSeek.

• A case study shows the Student refining teaching. On the Qwen HTML-finance family, the Educator proposes deriving reported values from other entries. The Student's initial implementation flags 4 metrics where only one is wrong; after inspecting output it replaces generic arithmetic with financial relations and reverse derivations, isolating the actual error at line 210 (the erroneous report entry). The final Skill passes 4/4 checks on an unseen cash-flow report.

• The benchmark was curated, not scraped. 35 families and 408 cases were produced from 37 candidate families and 444 cases, with 5–17 cases per family (35 anchors and 373 non-anchor cases). Reference solutions pass all 2,807 checks (3–34 per case).

Methodology in Plain English

The authors start from a simple observation: a successful task attempt tells you a procedure worked once, not whether it works under variation. So they separate the roles. A Student model (the same model that will later use the Skill) is the only agent that touches the environment, runs experiments, and writes the Skill package — a SKILL.md file plus optional scripts, references, and assets. An Educator model has read-only access to the Student's recent reasoning, tool results, and the live package, and can offer a lesson or stay silent. The Student can push back, test the suggestion, and decide what to keep.

Two schedules control when the Educator speaks. Fixed-Window SSL reviews after every g completed Student tool calls (g = 10 in the main condition, with g = 5 and g = 20 in the schedule comparison). Issue-Guided SSL instead tracks persistent "learning issues," each holding a learning focus and an observable acceptance criterion; review only happens when an unresolved issue opens, changes, or reopens. Issues form a tree, so closing a child returns attention to the nearest unresolved ancestor.

Acquisition runs in 3 rounds. After each round the package is frozen and tested by a fresh solver on the reset anchor, and that feedback informs the next round. The final frozen package is then evaluated by fresh solvers on held-out family members, so nothing from the authoring session leaks into evaluation except the package itself. Private tests and reference solutions stay outside agent workspaces.

The benchmark was built by adapting tasks from six source benchmarks (Terminal-Bench, AgentIF-OneDay, SpreadsheetBench, CodeContests, FullStackBench, OfficeBench) and constructing families through two routes: extraction and adaptation, which groups compatible source cases under a shared specification, and seed expansion, which starts from one task and authors additional in-scope cases. Every case passed programmatic checks (Docker builds, reference solutions passing tests) and review by an expert other than its author.

Utility is family-macro Avg@3 on a 0–100 scale, with families weighted equally, the main study using 3 independent acquisition runs per family and condition, and uncertainty reported via paired 95% bootstrap intervals that resample families. The model pairs are qwen3.7-plus (Student/solver) with qwen3.7-max (Educator), deepseek-v4-flash-0731 with deepseek-v4-pro-0813, and kimi-k2.5 with kimi-k3, run through the Pi coding agent 0.81.1 harness with TypeScript 5.9.3 and Docker environments.

Why This Matters

Impact on research. The paper reframes Skill acquisition as a question of when guidance arrives and what evidence it is allowed to shape, rather than only how trajectories are distilled. It also separates two things earlier benchmarks conflate: Skill utility and Skill transfer. Benchmarking transfer through a public family specification plus held-out evaluation in fresh sessions gives a reusable template for measuring whether learned procedures generalize. The process analysis — linking lessons to next-call writing behavior and package change over ten calls — offers a way to study teaching as a mechanism rather than a black-box input.

Real-world applications:

• Enterprise agent platforms that ship reusable Skills to many sessions, where the choice of how much expert review to buy directly determines operating cost.
• Coding and DevOps agents doing recovery, migration, and configuration tasks, where the benchmark's Systems and Services families (for example DB WAL recovery, ACL inheritance, deterministic tarball) resemble real operational work.
• Finance and office automation agents that must follow accounting relations and document conventions across varying report formats.
• Scientific computing pipelines where numerical recipes must hold across varying parameters, such as ODE RK4 or audio STFT peak selection.

Industry relevance. The cost result is the practical hook: on the Qwen main cohort, Issue-Guided SSL matches or slightly exceeds the fixed schedule while cutting Educator tokens by 61.7% and total acquisition tokens by 16.9%, and it achieves 2.75 times the transfer gain per Educator token. Since the Educator is typically the more expensive model in a deployment, selecting reviews around unresolved issues rather than on a timer is a directly monetizable design choice.

Future Directions

• Reducing total acquisition tokens, not just Educator tokens. The paper reports that total acquisition tokens fall for Kimi but rise for DeepSeek under Issue-Guided SSL because of additional Student work; how to control Student-side cost is left open.
• Extending the schedule comparison. The utility-versus-cost comparison is run for g = 5, g = 10, g = 20 and Issue-Guided SSL; whether finer or adaptive windows change the ranking is not reported.
• Closing the transfer gap on hard domains. Services and Data Processing show the lowest Skill utilities in Table 1 (for example Issue-Guided SSL at 71.0 in Services and 75.9 in Data versus 94.6 in Office), and the paper does not report what limits transfer there.
• Broadening the model-pair study. Only three model pairs are evaluated, DeepSeek and Kimi use a single repeat on the lite subset versus three repeats for Qwen, and it is not reported how the protocol behaves when Student and Educator differ more sharply in capability.

Target Audience

Researchers and engineers working on LLM agents, agent memory, or skill libraries; benchmark designers interested in transfer evaluation; and practitioners deploying reusable Skills in production who need to decide how much expert-model supervision to pay for. Readers should be comfortable with agent tool-use loops, trajectory distillation, and held-out evaluation protocols.

Authors’ abstract

Agent Skills improve LLM agents by making task knowledge reusable across sessions. Learning transferable Skills requires evidence about a procedure and its conditions of use, which a successful task attempt can leave untested. We introduce Socratic Skill Learning (SSL), which interleaves expert teaching with Student practice. An Educator guides investigation while the Student remains the sole environment operator and Skill author, able to question, test, and adapt advice. To test reuse beyond the learning task, we construct SkillTransferBench, with 35 task families and 408 executable cases, pairing public variation specifications with held-out evaluation in fresh solver sessions. On the main 22-family, 155-case cohort, Fixed-Window SSL reaches 0.801 utility, versus 0.723 without a Skill, 0.761 for adapted Trace2Skill (trajectory distillation), and 0.751 for adapted SKILL-KD (contrastive distillation). Across three model pairs on shared families, interleaved guidance outperforms expert authoring and post-hoc teaching. Process analysis links guidance to Student writing and package revision; a case study shows the Student testing and refining a proposed procedure for later reuse. As demand for new lessons falls, the cost of fixed-window review grows. Issue-Guided SSL tracks unresolved questions across Skill revisions to select and focus review, reaching 0.805 utility with 61.7% fewer Educator tokens and 16.9% fewer total acquisition tokens than Fixed-Window SSL on the main cohort.