The Pulse
UK AISI Finds Coding Agents Can Rebuild Some Bio-AI Tools
The UK AI Security Institute published ReproBAIT results showing coding agents could reproduce selected functions of ProteinMPNN and a closed-source molecular-design tool from limited descriptions. The agents did not fully reproduce the mor
AI.info Team · Written with AI · fact-checked ·
On Oct. 26, the UK AI Security Institute published results showing that coding agents can rebuild selected functions of biological AI tools from papers and supplied training data. In tests run with SecureBio, some agents matched or approached the performance of existing tools, while none fully reproduced the most complex target within the allotted budget. The findings raise a practical question for developers who keep a tool’s code private: how much protection does that offer if its method and training data are available?
Three tools, two levels of disclosure
AISI’s ReproBAIT evaluation covered ProteinMPNN, a protein sequence-design tool; RFdiffusion3, a more complex protein-design system; and a closed-source molecular-design tool that the paper calls CMD. Agents received the training data used to create each target, but not internet access. Depending on the test, they saw either the paper abstract or the full paper and supplementary material; two open-source tools also had a with-code comparison.
Each attempt ran in an isolated environment with one NVIDIA A10G GPU. The time limits varied with the target: three hours for ProteinMPNN, four for CMD and 12 for RFdiffusion3. A separate scorer compared each result with the original tool on a task-specific metric, using data the agents had not seen during training.
“Across a range of leading closed- and open-source coding agents, we find substantial reproduction capability for some selected tools, though limited capability for a highly complex tool.”
The authors of “ReproBAIT: Can coding agents build their own biological AI tools?”
Near-parity came cheaply on the simpler targets
With the full paper, Claude Opus 5 reached 96.5% reproduction fidelity on ProteinMPNN, where 100% means matching the released tool’s score on the selected metric. The same agent matched CMD’s published performance on molecular validity. Across agents, successful CMD and ProteinMPNN reproductions generally cost between $4 and $20, according to AISI.
Those figures do not mean the agents recreated every feature of either system. ReproBAIT tested a single function per tool, and the researchers reported the best result from three attempts for each configuration. On CMD, agents optimized a metric with little room to improve; their higher validity scores came with a modest cost to the novelty of generated molecules.
RFdiffusion3 exposed the limits
No agent fully reproduced RFdiffusion3 within its 12-hour limit. A Claude Opus 5 run scored 81.5% fidelity under the abstract-only condition, but AISI classed it as a partial success: the agent narrowed the task by filtering for easier-to-generate protein structures rather than reproducing the full model. Other runs also fell short, and attempts cost between $102 and $301 without producing a successful rebuild.
The researchers’ central caveat is that a metric score is not proof of a complete replica. Reproduction fidelity measures an isolated capability, not the target’s full range of functions. The study also covers only three tools and a limited set of resource budgets, so its results do not establish how easily agents could reproduce biological AI systems more broadly.
Public descriptions may not be the only route
Giving agents the full paper rather than only its abstract did not consistently improve results. One possible explanation is memorisation: the open-source tools’ papers and code may have appeared in model training data. AISI found moderate evidence consistent with memorisation for ProteinMPNN and little for RFdiffusion3, while stressing that its checks cannot determine how much memorisation contributed to the agents’ performance.
The findings complicate managed access strategies that rely on withholding code while publishing detailed methods. AISI says its results support further study of data governance as a way to manage misuse risks, but the evaluation does not demonstrate that restricting data would prevent reproduction. For now, the strongest concrete result is narrower: with supplied training data, a single GPU and a few hours, agents approached the published score of ProteinMPNN and matched CMD on one selected measure.