Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

AI summary

Overview

Research area: Natural Language Processing — automated construction and optimization of "skills" (external text artifacts placed in an LLM's context at inference time) without model weight updates.

Technical level: Intermediate. The framework itself is described procedurally and is approachable, but the paper includes formal problem definition, a paired statistical acceptance test, and a theorem bounding selection error under proxy-distribution mismatch.

Scope: The paper introduces Prompt2Skill, a multi-agent framework that turns a bare natural-language task description into an optimized, model-specific skill using retrieved or synthesized proxy data, and evaluates it on four benchmarks across five target models.

Authors and affiliations as listed: Bo Ni (Vanderbilt University), Li Li (University of Southern California), Ryan A. Rossi (Adobe Systems), Franck Dernoncourt (Adobe Systems), Tyler Derr (Vanderbilt University). Code is released at https://github.com/Arstanley/Prompt2Skill.

What This Paper Is About

Skills are external artifacts — often human-readable markdown files — that a model consumes at inference time to gain procedural and domain knowledge it does not reliably exhibit on its own. Authoring them by hand is expensive, the resulting artifacts are not optimized for the specific model that consumes them (whose failure modes vary with version, scale, and training), and new tasks may fall outside existing skill libraries. Prior automated approaches such as Trace2Skill and SkillOpt still assume a curated, in-distribution labeled training set, which a deployed user typically does not have. Prompt2Skill addresses this by building a skill from a natural-language task description alone, retrieving or synthesizing its own training data and refining the skill in a closed loop of reflective edits, with no weight updates and no user-supplied data.

Key Contributions

1. A prompt-to-skill framework. The authors present Prompt2Skill, described as, to their knowledge, the first end-to-end multi-agent framework that turns a bare natural-language task description into a model-optimized skill, using agentic data discovery and synthesis followed by closed-loop refinement on the skill — with no weight updates and no user-supplied data.
2. A comprehensive empirical study. Across four domains and five open-source and commercial target models, the paper reports that Prompt2Skill improves over direct prompting by 10.8% on average and never significantly regresses, with ablations and case studies isolating each component.
3. A statistical acceptance procedure and a selection guarantee. Candidate skill edits are accepted only when they win a statistically significant paired comparison on a fresh held-out batch, and Theorem 1 bounds the gap between the exported skill and the best retained skill under proxy-distribution mismatch and independent selection sampling.
4. A matched-budget comparison against prior skill optimization. Using the same proxy data, the same initial skill, Qwen3-8B as the solver, GPT-5.5 as the editor, and a cap of 7,400 solver evaluations per task, the paper compares Prompt2Skill with SkillOpt.

Main Findings

• Consistent gains across target models (Table 1). Prompt2Skill achieves the highest average relative improvement for every target model: +29.1% for Qwen3-8B, +26.1% for Qwen3-32B, +93.5% for Llama-3.2-1B over its three reported benchmarks, +22.0% for Claude Haiku 4.5, and +24.3% for GPT-5.5. The paper states gains range from 22.0% to 29.1% for the Qwen and commercial models.
• Aggregate figures stated differently across sections. The abstract reports an average improvement of 10.8% across open-source and frontier models; the introduction calls it "an average of 10.8% points"; the conclusion reports average relative improvements of 36.1% over direct prompting and 83.0% over off-the-shelf skills, averaged across the 19 evaluated model–benchmark pairs. These stated numbers do not agree with one another in the paper.
• Best mean score in 10 pairs. Prompt2Skill achieves the highest mean score in 10 model–benchmark pairs, including all five solvers on SQuAD and all four benchmarks for Qwen3-32B, although it does not achieve the best score in every setting.
• Fixed skills can regress badly. The off-the-shelf skill reduces Llama-3.2-1B's SQuAD score from 0.207 to 0.048, while Prompt2Skill raises it to 0.492. The off-the-shelf baseline also lowers Qwen3-32B's overall average (−9.7%) and Claude Haiku 4.5's (−6.2%). The paper concludes that a skill's usefulness depends on the target model and task, and that fixed baselines are built without feedback from target-model executions.
• Llama-3.2-1B could not complete any SpreadsheetBench task. This benchmark is reported as N/A for that model.
• Matched-budget comparison with SkillOpt (Table 2). Prompt2Skill improves SearchQA EM by 2.93 percentage points (48.71 vs 45.79), and SQuAD EM/F1 by 1.20/0.61 points (82.50 vs 81.30 and 91.11 vs 90.50).
• Initialization sensitivity. Neither the default seed skill nor an off-the-shelf domain skill consistently outperforms the other across SearchQA and SQuAD for Qwen3-8B, Qwen3-32B, and GPT-5.5; the off-the-shelf seed helps SQuAD for Qwen3-8B but hurts GPT-5.5, suggesting the method does not depend on a particular seed.
• Optional demonstrations are genuinely optional. Direct prompting on SQuAD improves from 70.5% with zero examples to 79.0% with four, whereas Prompt2Skill on SearchQA stays nearly unchanged between one and four examples (61.1% versus 60.9%).
• No detected test leakage. An audit of archived proxy data found no lexical overlap flags or workbook-hash matches against the target tests.
• A formal selection bound. Theorem 1 shows that with probability at least 1 − δ, the skill selected on a frozen selection set is within 2(ε_dist + ε_eval) plus a term of sqrt(2 log(2K/δ)/n) of the best retained skill, where ε_dist bounds the total-variation distance between the target and proxy distributions and ε_eval bounds the evaluator mismatch.

Methodology in Plain English

Prompt2Skill has two agents and works entirely on top of a frozen target model.

Task setup. From the task description d (plus optional few-shot examples), an LLM operator produces two things: a task specification τ recording the input structure, task type, language, expected answer style, and evaluation criterion; and a short seed skill s₀ stating the task and required output format. The evaluation criterion defines an executable proxy metric, such as normalized exact matching, numeric equivalence, or answer containment.

The Data Agent. Because the real task distribution is never observed, this agent retrieves candidate sources through tools that include interfaces to the Hugging Face dataset catalog and Wikipedia, generates retrieval queries (including queries derived from a hypothetical description of a suitable dataset), and inspects the returned schemas and samples. It either adapts retrieved records into the task's input–reference format via a proposed conversion plan, or generates new items grounded in retrieved material when conversion is insufficient.

Validation. Adapted and generated items pass structural checks (empty inputs, missing references, malformed fields), a task-compatibility check against the specification, an answerability screen that rejects sources on which the seed-equipped model fails every sampled item, and, for synthesized items, a consistency check requiring the seed-equipped model or a directly prompted model to produce an accepted answer. Duplicate filtering then produces the item pool.

The Skill Agent. The pool is split into three roles: a reflection set for generating rollouts and failure examples, a fresh acceptance batch for comparing candidate skills against the incumbent, and a frozen selection set established at initialization that keeps the same membership throughout. Each round, the incumbent skill is executed on the reflection set, a reflector model proposes candidate edits from the observed failures, and each candidate is evaluated on the same fresh acceptance batch as the incumbent. A candidate is accepted only if it solves strictly more items than the incumbent solves alone and its paired statistic z (computed from wins and losses) is at least 1. Among qualifying candidates, the one with the largest net gain is chosen.

Selection and export. Every accepted skill is saved as a checkpoint. At the end, the initial skills and all previously accepted skills are compared once on the frozen selection set, and the highest-scoring entry is exported as SKILL.md. GPT-5.5 supplies the reflective feedback for the optimization runs in the experiments. Round budgets, stopping criteria, dataset splits, evaluation metrics, and prompt templates are deferred to the paper's appendices and are not included in the provided content.

Why This Matters

The work removes the assumption that a user must already possess a labeled, in-distribution training set before a skill can be optimized for their task — an assumption the paper argues collapses automated gains to roughly the level of a data-blind draft when the training set shrinks to a single example. It also reframes skills as artifacts that should be optimized per target model rather than authored once and shipped universally, supported by the observation that the same off-the-shelf skill helped some models and substantially hurt others.

Real-world applications:

• New-domain assistants with no training data. A team that has only a description of the behavior it wants can obtain a skill without collecting or labeling examples first.
• Document and passage question answering. The reading-comprehension and factual QA settings (SQuAD, SearchQA) map directly onto internal knowledge-base and Wikipedia-style answering over a passage.
• Spreadsheet and data manipulation. SpreadsheetBench targets automating multi-step spreadsheet edits, a common enterprise back-office task.
• Mathematical and competition reasoning. The AIME setting targets models that must solve quantitative problems with correct final answers, relevant to tutoring and analytical tooling.

Industry relevance: Because the method requires no weight updates and no user-supplied data, it fits deployment workflows where retraining is infeasible or a model is accessed only through an API, and where a model's failure modes shift with version, scale, and training. The reported regressions from fixed skills suggest that shipping a single hand-authored or LLM-authored skill across a model fleet carries real risk.

Future Directions

• Reducing proxy-to-target mismatch. The paper's own bound depends on ε_dist (the total-variation gap between the proxy data and the target distribution) and ε_eval (evaluator mismatch); no experiments are reported that measure how large these gaps actually are or how to shrink them.
• Establishing reference correctness. The paper notes that the validation screens provide practical evidence of usability and that solver agreement alone does not establish reference correctness or representativeness of the target distribution — an open problem the authors leave unaddressed.
• Reconciling and standardizing the reported gains. The abstract, introduction, per-model table entries, and conclusion state different aggregate improvements, and the paper's section on baselines says there are four while enumerating three; a clearer, unified reporting protocol would be a natural next step.
• Scaling and transferring skills. The evaluation covers four benchmarks and five target models, with some combinations missing (Llama-3.2-1B could not complete SpreadsheetBench); whether optimized skills transfer across model versions, scale, or related tasks, and whether refinement budgets can be reduced below 7,400 solver evaluations per task, remain open questions.

Target Audience

Researchers and practitioners working on prompt optimization, agent skills, and inference-time adaptation for LLMs; engineers who need domain-specific model behavior without fine-tuning or labeled data; and readers interested in NLP AutoML pipelines that retrieve data and build artifacts rather than train weights. Readers wanting the full algorithmic detail — round budgets, stopping criteria, dataset splits, metrics, prompt templates, and the rest of the Theorem 1 proof — will need the appendices, which are not included in the available content.

Authors’ abstract

Skills are external artifacts that Large Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present Prompt2Skill, a framework that builds skills from natural-language task description alone. From the prompt, the system derives a task specification, discovers or synthesizes datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of 10.8 across open-source and frontier models.