Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

AI summary

Overview

Research area: Autonomous AI research agents; knowledge distillation from software repositories into reusable agent "skills" (AI-for-AI).

Technical level: Intermediate — the paper mixes formal notation (task tuples, skill graphs) with a system description and benchmark tables, but the core ideas are explained in plain operational terms.

Scope (1 sentence): The paper proposes DisCo, an agent that distills large, human-oriented ML repositories and papers into compact, verified, agent-readable skills, and shows that attaching these skills to a fixed research agent (Codex with a GPT-5.5 backbone) improves scores on four research benchmarks (arXiv:2609.02749v1, published 2026-09-02; code at https://github.com/VectorSpaceLab/AREX-Skill).

What This Paper Is About

Autonomous research agents are built from two parts — a model backbone and a harness for planning, execution, memory, and verification — but neither part carries the domain-specific know-how that separates "knowing a method" from "making it work." The paper calls this missing layer operational knowledge, which today exists only inside repositories and papers written for human readers and is far too large to load during a task. The goal is to automatically distill that material into compact, verified skills that an agent can discover, load on demand, and reuse across tasks instead of rediscovering everything through trial and error.

Key Contributions

1. Names the missing layer. The paper identifies operational knowledge as a third component of research agents alongside the model and the harness: the harness governs how an agent researches, while operational knowledge determines what it knows when research begins.
2. Presents DisCo, a skill-powered research agent that both creates skills and researches with them. Distillation runs in two complementary forms — task-agnostic (from a source such as a repository or paper) and task-oriented (from a concrete task) — and no skill is admitted without verification.
3. Builds the AREX-Skill Library by scaling DisCo across the open ecosystem: 5,000+ verified skills distilled from 1,000 widely used ML repositories, organized into 20 areas and 178 capability families and exposed through a library-level router. The current repository snapshot contains 5,353 skills across 1,000 repository graphs, with 2,209 exact repository-to-area-to-family assignments and 700 repositories appearing in more than one family.
4. Evaluates distilled skills on MLE-bench, PaperBench, FrontierCS, and PassNet under a matched GPT-5.5 backbone, research harness, and downstream execution budget, observing gains on all four, up to 134.3% on MLE-bench.

Main Findings

• Skills substantially raise MLE-bench scores under a fixed setup. Adding distilled skills to Codex (GPT-5.5) raises the overall Any-Medal score on the full 75-competition suite from 31.11% to 72.89%, a gain of 41.78 percentage points or a 134.3% relative improvement.
• Gains appear across every difficulty tier. Low rises by 43.94 points (42.42% to 86.36%), Medium by 37.72 points (31.58% to 69.30%), and High by 48.89 points (13.33% to 62.22%, a 366.8% relative improvement, or 4.67 times the no-skill score).
• The gain does not require a new harness. Skill-equipped Codex surpasses the strongest public baseline in the table (Famou-Agent 2.0 with Gemini-3-Pro-Preview at 64.44% ± 1.18) by 8.45 points overall, and by 6.06 points on Low, 5.26 points on Medium, and 15.55 points on High — using vanilla Codex with added distilled skills and no custom execution harness or modified control loop.
• The advantage grows with task difficulty. The largest gaps appear on High-difficulty tasks, both against Codex without skills (+48.89 points) and against the strongest public baseline (+15.55 points).
• PaperBench improves broadly. Distilled skills raise the average replication score across 20 papers from 29.45% to 39.59% (+10.14 points, a 34.4% relative improvement), improving 18 of 20 tasks and degrading 2.
• The biggest PaperBench gains are on low-baseline tasks. ftrl rises from 1.50 to 17.17 (an 11.4× increase), rice from 7.94 to 48.51 (6.1×), and what-will-my-model-forget from 9.35 to 30.45 (3.3×). Tasks already scoring moderately high, such as all-in-one (52.93) and bam (56.65), gain less (+1.77 and +2.18).
• A small number of PaperBench tasks regress. sample-specific-masks falls from 57.11 to 52.04 (−5.07) and stay-on-topic from 32.31 downward by −4.52 (the truncated content does not show the final value).
• FrontierCS and PassNet also improve, by 9.2% and 14.0% respectively over the same agent without skills; the truncated content does not include the per-task tables for these two benchmarks.
• Skills are verified before admission. Each candidate graph is checked using assertion-backed cases and safe repository-native examples, tests, CLI checks, tiny-fixture checks, or smoke scripts; failures attributed to the graph trigger local repair and reruns, and any surviving gap is recorded in a construction record rather than hidden.
• Construction cost is reported. Repository graphs are built with GPT-5.5 and GPT-5.6-sol at xhigh reasoning effort, at an average allocation of about $40 per repository; skill construction and benchmark execution use separate per-task budgets, so the one-time construction cost is not counted in either run-time condition.

Methodology in Plain English

Formalizing the gap. The paper writes a research task as a tuple τ = (q, D, E, g) — problem, given data/material, environment and budget, and target — and describes an agent as A = (M_θ, H): a model and a harness. It then adds a third term, K, the operational knowledge made available as explicit operating context, giving A_res = (M_θ, H, K).

What a skill is. K is instantiated as a set of skills. Each skill has three layers: SKILL.md (the knowledge interface, the only layer read up front, stating goals, key concepts, tool usage, pointers, worked examples, and known failure modes), references/ (the knowledge substrate, holding API documentation, algorithmic detail, and parameter configurations, loaded only when needed), and scripts/ (the execution interface, executable wrappers the agent invokes rather than reimplements). references/ and scripts/ turn knowledge into capability; SKILL.md turns capability into usage policy. Skills distilled from one source are organized as a skill graph G = (S, L) with an entry skill that routes to component skills; progressive disclosure means the agent opens only the branch its task needs.

Four-stage distillation. Every run follows the same pipeline: scope (decide which capabilities matter) → ground (gather evidence) → construct (assemble a candidate graph in the three layers) → verify (accept a graph plus a construction record). In task-agnostic distillation the anchor is a source such as a repository or paper, built ahead of time and reusable by any later task; scoping is Source Understanding then Capability Identification, grounding is Knowledge Extraction, construction is Tool Encapsulation then Skill Packaging, and verification is Skill Verification. In task-oriented distillation the anchor is a task; scoping is Task Decomposition then Capability Gap Analysis, grounding is Source Discovery (searching for missing material rather than selecting it), and construction is Skill Generation.

Two modes, one agent. In creator mode DisCo performs distillation and writes accepted graphs into the AREX-Skill Library; in researcher mode it solves a task using K drawn from that library. The modes are deliberately asymmetric in cost: creation is paid once per source and amortized, while research pays only for what a task opens. Creating a skill requires no change to the backbone or the harness.

Library organization. Repository graphs are indexed by a two-level taxonomy of 20 areas and 178 capability families, built by freezing a short summary per repository (excluding stars, URLs, and pre-existing category fields), inducing a tree with an LLM-assisted pipeline evaluated on 100 stable batches of 10 repositories, and then assigning each verified graph to exact area-to-family paths with mandatory rationale, evidence, and confidence — rejecting keyword-only, dependency-only, optional-integration, and example-only matches. A router narrows a request from area to family to repository graph.

Experimental controls. Codex is the harness and GPT-5.5 with xhigh reasoning effort is the backbone in every condition. The only variable is whether DisCo-distilled skills are attached. For MLE-bench, a dedicated graph is built for each of the 75 competitions from web-searched sources, excluding the original competition webpage and competition-specific content. For PaperBench, skills are distilled from papers cited in each target's related-work section and their repositories, excluding the target paper and its released artifacts; this produced 636 paper-derived skills from 153 source papers. MLE-bench and PaperBench runs use the benchmarks' held-out/official graders. FrontierCS uses one shared recovery-oriented graph across its 188 Agent Track tasks, run through Harbor with a 5-hour budget per task and a container limited to 2 CPUs and 4 GiB of RAM. PassNet uses one benchmark-level graph covering FX-graph inspection, pattern matching, semantics-preserving rewrites, Triton implementation, and performance diagnosis, evaluated on NVIDIA A100-SXM4-40GB.

Why This Matters

Impact on research. The paper argues that progress on autonomous research agents has come almost entirely from stronger backbones and better harness engineering, leaving domain know-how external. It shows that a separate, reusable knowledge layer can be built from already-public material and attached to a conventional agent, producing large gains without changing the model or the control loop — and that the layer is portable across compatible harnesses because skills are agent-readable files.

Real-world applications.

• ML competition and benchmark automation: task-specific graphs built per competition (75 for MLE-bench) let an agent skip misconfigured runs and reach competitive solutions faster.
• Paper reproduction: distilled skills from related work and their repositories improve replication of published results (18 of 20 PaperBench targets improved here).
• Scientific and engineering software onboarding: turning repositories such as sentence-transformers, AlphaFold, and vLLM into skill graphs gives agents usable data-preparation, training, inference, evaluation, serving, troubleshooting, and maintenance procedures.
• Compiler and performance optimization: PassNet-style graphs package FX-graph inspection, semantics-preserving rewriting, Triton implementation, and performance diagnosis into reusable operating context.

Industry relevance. The library is a curated snapshot of commonly used ML software organized by a capability taxonomy (20 areas, 178 families), with a router that lets an agent load only the relevant branch. The reported creation cost of about $40 per repository is a one-time, amortizable expense, which makes the approach plausible for teams that want to give coding or research agents operational knowledge of their own stack.

Future Directions

• Extend paper-derived distillation beyond the current scope. The paper notes the same workflow can be extended to additional papers "as verification budget permits"; only 20 PaperBench targets and 153 source papers were covered.
• Understand and reduce regressions. Two of 20 PaperBench tasks scored lower with skills; the paper does not report an analysis of why, leaving room to characterize when injected skills mislead.
• Scale and refresh the repository snapshot. The 1,000-repository collection is described as a curated snapshot rather than an exhaustive partition of the ecosystem, and source material drifts with every release.
• Grow the taxonomy and router. The current 20 areas and 178 capability families, with 700 repositories appearing in more than one family and some graphs left unclassified when no exact family is supported, raise questions about coverage as the ecosystem changes.
• Report and interpret the FrontierCS and PassNet details. Headline gains of 9.2% and 14.0% are stated, but the truncated content does not include their per-task results or the trade-offs behind them.

Target Audience

Researchers and engineers working on autonomous ML research agents, agent harness design, and AI-for-AI systems; practitioners who want to package internal or open-source tooling into reusable agent skills; and readers interested in knowledge distillation from code repositories and papers into compact, verified, loadable artifacts. A working familiarity with LLM agents, benchmark evaluation, and basic ML tooling is helpful, but the central argument is accessible without deep mathematical background.

Authors’ abstract

Autonomous agents are beginning to carry out machine-learning (ML) research end to end. These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research agent that creates skills and uses them during research. Its distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills. These gains come from adding distilled operating context under that fixed setup.