CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

AI summary

Overview

Research area: Automated Scientific Discovery (ASD) and code-generating AI agents, applied to the materials science domain.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, retrieval-augmented generation, and benchmark evaluation, but the system design and findings are explained concretely.

Scope: CodeDistiller is a system that automatically converts large collections of scientific GitHub repositories into a vetted library of working, domain-specific code examples that automated discovery agents can retrieve and combine, replacing manual curation of such libraries.

What This Paper Is About

Automated scientific discovery agents can generate and debug code to run computational experiments, but they are limited to what they learned during training, so they either mutate a small number of hand-crafted experiment examples or work from parametric knowledge alone. CodeDistiller addresses this by automatically distilling hundreds of scientific GitHub repositories into executable, debugged examples of each repository's core functionality. The result is a code library that an agent can draw on to perform tasks beyond its built-in knowledge, without a human writing the examples.

Key Contributions

1. CodeDistiller, a system that automatically converts GitHub repositories in specialized scientific domains into debugged, working code examples suitable for incorporation into automated scientific discovery systems.
2. A materials science evaluation across 250 repositories, showing the best and worst performing base models successfully distill between 26% and 74% of repositories, with different price/performance tradeoffs.
3. A downstream evaluation showing that a discovery system augmented with a CodeDistiller-built library generates more accurate, complete, and scientifically sound output than a baseline model with only general materials science code examples.
4. A characterization of agreement between LLM-as-a-judge and domain-expert judgments in materials science, showing moderate agreement for downstream discovery quality in an A/B testing paradigm, but mixed agreement depending on the model when evaluating the code distillation task itself.

Main Findings

• Best model reaches 74% functional examples. Under domain-expert judgment, Claude Sonnet 4.5 produced examples with correct functionality for 74.1% of the 250 materials science repositories. GPT-5 reached 60.5%, and GPT-OSS-120B reached 25.9%.

• Automatic judging overestimates quality, most severely for the weakest model. The LLM-as-a-judge "Successfully Completed" metric reported 61.6% for GPT-OSS-120B, 70.4% for GPT-5, and 75.6% for Claude Sonnet 4-5. Domain-expert "Correct Functionality" dropped these to 25.9%, 60.5%, and 74.1% respectively, so the gap is large for GPT-OSS-120B and modest for Claude 4.5.

• Claude Sonnet 4.5's manual scores are identical across three of four metrics. Code Executed without Error, Demonstrates Repository Functionality, and Correct Functionality were all 75.6%, 75.6%, and 74.1% respectively; for GPT-OSS-120B the first two were 29.6% and the third 25.9%; for GPT-5 they were 69.0%, 69.0%, and 60.5%.

• Higher performance costs substantially more. Average cost on successful cases was $0.09 for GPT-OSS-120B, $0.70 for GPT-5, and $1.71 for Claude Sonnet 4.5 — the average Claude-generated example cost 19 times more than the GPT-OSS version.

• Runtimes and debug effort are broadly similar across models. Successful examples took an average of 13.8 minutes (GPT-OSS-120B), 20.3 minutes (GPT-5), and 19.0 minutes (Claude Sonnet 4-5), requiring 2.4, 2.2, and 1.9 debug iterations respectively. Unsuccessful cases were costlier in both time and API spend, with GPT-5 averaging 47.0 minutes and $1.68.

• Downstream discovery improves with the distilled library. In a blind A/B test over 50 discovery problems run through CodeScientist, the baseline system (generic materials science examples only) was preferred in 18% to 24% of cases across accuracy, completeness, and soundness, while the CodeDistiller-augmented system was generally preferred in more than half of cases, with roughly a quarter of problems tied.

• Domain-expert and LLM-as-a-judge ratings agree moderately on downstream quality. Cohen's Kappa was 0.77 for accuracy (approaching strong agreement), 0.70 for soundness, and 0.62 for completeness (bordering weak agreement).

• The distilled examples enable qualitatively better science. Documented cases include using the actual Tox21 dataset with 6,258 real compounds across 12 toxicity assays instead of synthetic fabricated data (20 hand-picked molecules replicated), and using CHGNet — a materials-specific ML potential trained on DFT data — instead of a generic Lennard-Jones potential, producing volume changes of -16% to +75% rather than unphysical collapses of 80-93%.

• Scale of the repository pool. Searching the GitHub API for active, permissively licensed repositories importing at least one of 30 popular materials science Python libraries yielded 3,802 unique repositories, randomly subsampled to 250 for the evaluation.

Methodology in Plain English

CodeDistiller runs in two broad phases. First, it performs static analysis of a repository: each file is individually fed to a prompt that classifies it (code, documentation, scripts, data, or other), subdivides three of those classes into finer types such as existing examples, instructions, and entry points, assigns a relevance score from 1 to 5, and records metadata such as whether the file mentions GPUs or contains critical configuration instructions. This matters because repositories can contain hundreds of nested files while only a few demonstrate core functionality, and because the base model's context window is limited.

Second, the highest-ranked files plus a description of the repository's core purpose are given to a code generation system built on a modified version of the CodeScientist agent. It produces four things: executable Python code, Python library requirements, a bash runscript with Conda environment setup, and metadata including a description, inclusion/exclusion criteria, resource requirements (CPU cores, GPUs, RAM, disk), and whether the code is stand-alone or requires user interaction. The code runs in an Ubuntu cloud container with instrumented output — a timestamped log file, a JSON results file, raw stdout/stderr logs, and encouraged figures — and an LLM-as-a-judge decides whether it worked. If problems are found, the code is reflected on using the current version and logs, then re-executed, repeating until judged successful or until a limit (typically 8 debug iterations) terminates the run as unsuccessful.

Evaluation used three base models at different price points: GPT-OSS-120B, GPT-5, and Claude Sonnet 4.5. File classification used cheaper models in the same families (for example GPT-5-mini and Claude Haiku 4.5), except GPT-OSS-120B, which was cheap enough for the whole pipeline. A materials science domain expert with a graduate degree manually reviewed a subsample of 50 repositories that all three models had marked as successfully completed. For the downstream test, 60 discovery problems targeting the primary purpose of 12 expert-filtered repositories (5 problems each) were generated with Claude Sonnet 4.5 and given to CodeScientist in baseline and experimental configurations, with up to 15 debug iterations, 6 hours total runtime (capped at 60 minutes per iteration), and up to $5 USD of LLM cost per run. Each task was run twice, and the first successful run was used. Outputs were rated blind on accuracy, completeness, and soundness in a chain-of-thought A/B paradigm, with the LLM-as-a-judge rating each problem twice to counterbalance presentation order.

Why This Matters

CodeDistiller targets a bottleneck in automated scientific discovery: agents can only run experiments they can code, and their coding ability is bounded by training data. By turning the enormous, messy supply of scientific GitHub repositories into vetted, runnable examples, the system lets discovery agents expand into specialized methods without a human hand-writing each example. The authors note that the six discoveries made by CodeScientist depended on a library built through a combination of manual generation and expert curation, whereas CodeDistiller allows such libraries to be augmented automatically.

Real-world applications:

• Materials science research automation. Running computational experiments on magnetic symmetry, cheminformatics, and condensed matter using established, peer-reviewed libraries such as pymatgen, ASE, LAMMPS, and PyCalphad rather than reimplementing calculations from scratch — the paper's own example shows a discrepancy where a from-scratch atomic size difference value for AlTiVNb was 3.60% versus 5.428% from the library-based solution.
• Reproducibility and benchmarking pipelines. Automatically producing minimal working examples of a repository's core functionality gives a cheaper way to smoke-test whether published code actually runs, which is directly relevant to the low success rates reported on related benchmarks such as SUPER (best performance 16%) and ResearchCodeBench (best model under 40%).
• Lower-cost evaluation of discovery systems. The moderate agreement between LLM-as-a-judge and domain experts (Kappa 0.62 to 0.77) suggests automated ratings could serve as an approximate proxy for expert ratings, which the authors note took several weeks to collect for this study.
• Tooling for general-purpose coding agents. The authors frame CodeDistiller as usable either as a purpose-built preprocessing agent that builds code libraries, or as a tool or subagent that more general agents such as Claude Code or Codex could call during execution.

Industry relevance is primarily for groups building AI research assistants and autonomous experimentation platforms in computationally intensive scientific and engineering domains, where reliable access to domain-specific, runnable code is the difference between a plausible-looking report and a scientifically grounded result.

Future Directions

• Extending beyond materials science. The paper explicitly limits its evaluation to one domain and notes that transferring to other domains may encounter domain-specific obstacles, including software and data availability, that could affect both distillation and downstream discovery performance.
• Resolving the agreement gap on the distillation task itself. LLM-as-a-judge showed moderate agreement with experts on downstream quality but mixed agreement, depending on the model, when evaluating code distillation. Understanding when automated judges can be trusted for the code-generation step remains open.
• Improving repository identification. The authors estimate that only about half of the repositories collected by their import-based search are directly related to materials science; the rest merely import a library that materials science happens to use. Better targeting would improve precision.
• Comparing purpose-built versus general-purpose agents. The authors leave open whether a purpose-built agent like CodeDistiller or a general-purpose agent is better for this class of task, noting the difficulty of controlled comparisons given differing inputs, budgets, models, architectures, and highly variable output.
• Improving the weakest models rather than only the strongest. The steep cost-performance tradeoff — with the strongest model costing 19 times more per successful example than the cheapest — raises the question of whether cheaper pipelines can be made competitive.

Target Audience

Researchers and engineers working on automated scientific discovery, LLM coding agents, and retrieval-augmented code generation will get the most from this paper, along with materials scientists interested in how their domain's open-source tooling is being consumed by AI systems. It is also relevant to evaluation researchers, since a substantial part of the paper concerns how well LLM-as-a-judge ratings track domain-expert judgments and where the two diverge. Readers need some familiarity with agent pipelines and benchmark methodology, but the system description and the results tables are self-contained enough for a motivated non-specialist.

Authors’ abstract

Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by the code they can reliably generate from parametric knowledge alone. As a result, current systems either mutate a small number of manually-crafted experiment examples, or operate solely from parametric knowledge, limiting quality and reach. We introduce CodeDistiller, a system that automatically distills large collections of scientific Github repositories into a vetted library of working domain-specific code examples, allowing ASD agents to expand their capabilities without manual effort. Using a combination of automatic and domain-expert evaluation on 250 materials science repositories, we find the best model is capable of producing functional examples for 74% of repositories, while our downstream evaluation shows an ASD agent augmented with a CodeDistiller generated library produces more accurate, complete, and scientifically sound experiments than an agent with only general materials-science code examples. We also evaluate LLM-as-a-judge ratings against domain-expert ratings in an A/B testing paradigm, finding moderate agreement and suggesting that inexpensive proxy metrics may be feasible for evaluating scientific discovery systems at scale.