MoLRSI: A Recursive Self-Improving AI agent for Molecular Discovery

AI summary

Overview

Research area: AI agents for molecular discovery, combining automated machine-learning workflow engineering with persistent memory and tree search; evaluated prospectively in antibiotic screening and peptide–MHC binder design.

Technical level: Advanced. The paper assumes familiarity with graph neural networks, protein design tools (RFdiffusion, ProteinMPNN, AlphaFold2), Monte Carlo tree search, and wet-lab screening readouts such as MIC and enrichment factors.

Scope (one sentence): The paper introduces MolRSI, an agent that searches over molecular-discovery workflows with experience-guided Monte Carlo tree search and three kinds of persistent cross-task memory, then validates the resulting workflows experimentally.

What This Paper Is About

Designing a computational molecular-discovery workflow means choosing among many molecular representations, model architectures, training strategies and selection criteria, and those choices interact in ways that vary by biological target. Automation alone does not solve this: existing agents (Biomni, Claude Science, STELLA, Codex, Claude Code) can execute or compose protocols, but each campaign tends to start from scratch, so lessons from one project do not shape the next. MolRSI's goal is to make the agent's search policy itself improve through use — recording which workflows performed well, which modifications helped, and which execution failures were repaired — so that later tasks are approached more effectively.

Key Contributions

1. A recursive self-improvement (RSI) architecture for workflow development. MolRSI couples experience-guided Monte Carlo tree search (MCTS) over candidate workflows with persistent memory, so that each search both solves the current task and changes how future tasks are searched (the policy shifts from π_t to π_t+1).

2. Three explicit memory stores. Solution memory holds workflow code, configurations, execution status and development scores; refinement memory holds proposed modifications, their rationale, and before/after scores (retained even when performance declines or execution fails); execution memory holds runtime, GPU time, GPU memory, error logs and the success or failure of each attempted repair.

3. An Explore Agent that turns literature and repositories into validated skills. The agent formulates queries, writes reusable skill descriptions (purpose, dependencies, inputs/outputs, execution steps), and runs a usage example in an isolated environment, repairing or discarding skills that fail — yielding a library of 110 skills (55 models, 48 methods, seven tools or resources) for the antibacterial work and 65 executable skills for protein design.

4. Prospective experimental validation in two settings with competing objectives. In antibiotic discovery the agent produced a screening workflow with a 15.9% hit rate (17 of 107 compounds) versus 8.7% (21 of 241) for the prior expert-led workflow; in pMHC design it produced eight experimentally validated binders where Claude Science produced none.

Main Findings

• Highest benchmark accuracy in antibacterial activity prediction. Without RSI, MolRSI achieved the highest mean held-out test AUPRC among Biomni, Claude Science, STELLA, Codex and Claude Code across six antibacterial tasks (one per bacterial species), ranking first on four tasks and second on the other two, using five similarity-based data splits.

• Experience improves search, not just reuse. Drawing on 25 source tasks (22 Therapeutics Data Commons tasks plus three cytotoxicity tasks for HepG2, HSkMC and IMR-90 cells, with the six antibacterial tasks excluded), RSI improved mean test AUPRC on all six antibacterial tasks. On E. coli and N. gonorrhoeae, the first workflow evaluated with experience already outperformed the final workflows of every baseline after 20 search iterations.

• Zero-search transfer can beat search from scratch. Directly reusing stored workflows (retaining model fitting and validation, but doing no new search) outperformed search without RSI on S. aureus, A. baumannii and K. pneumoniae, and gave the best mean test AUPRC of the three conditions on K. pneumoniae.

• More experience yields better transferable workflows. Averaging over all memory subsets of each size (2,000 sampled subsets per pool size), mean test AUPRC rose from approximately 0.32 with two source tasks to approximately 0.37 with all 25, exceeding the 0.361 reached by search without RSI after 20 iterations.

• Improved prospective antibiotic hit rate. Screening approximately 12.1 million compounds with the same training data and library as the prior expert study, MolRSI selected 107 compounds for testing; 17 had MICs of 32 μg/mL or lower against methicillin-susceptible S. aureus, a 15.9% hit rate versus 8.7%, an approximately 82% relative increase.

• Hits with separation from cytotoxicity. C3 and C32 inhibited MRSA with MICs of 1 and 8 μg/mL, while CC50 values ranged from 6.4 to 17.5 μg/mL for C3 and 24.2 to 54.7 μg/mL for C32 across three human cell lines. They ranked 28th and 5th by predicted antibacterial activity among tested compounds, and hits spanned multiple regions of chemical space.

• Better activity prediction on a shared test set. On the 271-compound held-out S. aureus set from the previous study, MolRSI reached AUROC 0.828 versus 0.590 and AUPRC 0.300 versus 0.090 for the prior predictor.

• Stronger binder prioritization on 14 miniprotein targets. MolRSI achieved the highest macro enrichment factor at both cutoffs: 8.01 at EF@1% (next-best agent 7.62) and 7.29 at EF@2% (next-best 6.71).

• RSI drives pMHC ranking performance. On the held-out MART-1 target, MolRSI reached EF@1% of 3.65 and EF@2% of 5.55; no baseline identified an experimentally confirmed binder in the top 1%, and the highest baseline EF@2% was 1.85. The same system without experience from the 14 miniprotein tasks scored EF 0.00 at both cutoffs.

• Design optimization gains. A 30-round search over generation methods, holding the ranking procedure fixed, raised the mean score of the top 5% of candidates to 6.76 for MART-1 and 7.56 for SARS-CoV-1 — improvements of 66% and 28% over initial configurations.

• Eight experimental hits, none for Claude Science. In E. coli phage display with two rounds of selection, MolRSI yielded five MART-1 hits among 251 evaluable designs and three SARS-CoV-1 hits among 266; Claude Science yielded none among 288 and 267. Removing the learned ranking procedure yielded no hits among 479 evaluable designs.

• Computational score tracks experimental success. When evaluable designs from all three groups were ranked together, all five MART-1 hits fell within the highest-scoring 9% of 767 designs, and the three SARS-CoV-1 hits fell within approximately the highest-scoring quarter of 784 designs.

Methodology in Plain English

Each task is specified by a biological problem, available data, an evaluation criterion and a computational budget. An Explore Agent then reads relevant papers and code repositories and turns each resource into a reusable, executable "skill," testing it in an isolated environment and repairing or dropping it if it fails.

Candidate workflows are assembled from these skills and improved through a tree search. Workflows sharing a model or computational approach are grouped into families. In each iteration the agent (1) picks a family using current-task performance, an exploration bonus for under-visited families and a historical transfer score from memory; (2) samples a parent workflow, favouring better performers and penalizing those that have already produced many candidates; (3) retrieves past modifications and their outcomes to propose a change, with a stated rationale, and checks the proposal before running it; then executes and evaluates the revised workflow (the best-scoring workflow within budget is returned) and (4) writes the results back into memory.

The key design choice is that memory outlives the task. Performance scores, the measured effect of each edit, and the fixes for each execution error remain available, so the next search starts from a different set of priorities. Three comparison conditions separate the effects: search with RSI, search without RSI (current-task experience only), and zero-search transfer (reuse a stored workflow with only model fitting and target-data validation).

For antibiotic work, the agent balanced predicted activity against MRSA with predicted cytotoxicity in HepG2, HSkMC and IMR-90 cells, added structural filters excluding specified antibiotic cores and compounds similar to known antibiotics, and selected candidates across structural clusters. For pMHC work, the ranking workflow was developed and validated on two gp100 datasets with MART-1 held out, then used to steer generation-method search against MART-1 (ELAGIGILTV) and SARS-CoV-1 (GLMWLSYFV) pMHCs, each with a peptide-swapped off-target. Designs were compared using MolRSI's fixed ranking score and using the summed ipSAE min z-scores of Claude Science's three prediction models, with z-scores computed per target and prediction model. Hits in screening required on-target log2FC greater than 4 that also exceeded the highest measurable off-target enrichment.

Why This Matters

Impact on research. The paper reframes agent evaluation: instead of asking only whether an agent produced a good final workflow, it asks whether the agent's search policy improved. The with-RSI, without-RSI and zero-search-transfer comparisons give a template for attributing gains to inherited configurations versus further development, and the results show that experience can be transferred between related molecular tasks. It also links computational scores to prospective wet-lab outcomes rather than retrospective benchmarks alone.

Real-world applications.

• Antibiotic discovery against drug-resistant pathogens: a screening pipeline that jointly optimizes activity, mammalian cytotoxicity and structural novelty, with experimentally confirmed hits including MRSA-active compounds C3 and C32.
• Cell therapy and protein therapeutics: pMHC-binding miniproteins that discriminate a presented disease peptide from closely related off-targets, relevant to chimeric antigen receptor designs.
• Prioritization for any costly experimental campaign: the ranking workflow improved enrichment on 14 miniprotein targets and was reusable on unseen pMHC targets.
• Reusable automation infrastructure: validated executable skills and repair records that reduce repeated debugging across projects.

Industry relevance. Discovery teams pay heavily for each wet-lab round; a workflow that raises hit rate from 8.7% to 15.9% on a 107-compound test set, or removes an entire failed screening campaign (eight hits versus zero), directly changes the economics of lead generation. The persistent-memory design also matters for organizations running many parallel campaigns, where institutional knowledge currently lives with individual scientists.

Future Directions

• Closing the loop with experimental outcomes. The Discussion proposes feeding laboratory successes and failures back into policy updates, so wet-lab evidence shapes which workflows receive further exploration.
• Benchmarks for policy adaptation. The authors call for benchmarks that measure how effectively an agent adapts its search strategy across successive tasks, not just final-task performance.
• Robustness as targets and criteria change. Whether policy updates stay useful when biological targets and evaluation criteria shift is left open; the observed variation across experience pools makes the choice of contributing tasks a live question.
• Breadth beyond the two settings. The paper reports antibiotic screening and pMHC binder design; generalization to other discovery problems, and the relative value of skill libraries versus memory, are not established.

Target Audience

Machine-learning researchers building scientific agents and automated ML pipelines; computational biologists and protein engineers interested in binder design and virtual screening; drug-discovery teams assessing whether agentic workflows justify wet-lab spend; and research leaders interested in how systems can accumulate institutional knowledge across projects. Readers without background in molecular representation or protein structure prediction will find the experimental results accessible but the methodological sections dense.

Authors’ abstract

Molecular discovery requires navigating vast combinatorial search spaces while balancing competing biological objectives. AI agents can automate expert-defined molecular design protocols, but automation alone does not address how to optimize workflows for new tasks or improve their development through experience. Here we introduce MoLRSI, an AI agent that combines systematic optimization of workflows and model architectures with persistent cross-task memory to recursively improve its search policy for molecular discovery. Workflow performance, refinement outcomes, and execution feedback guide subsequent searches, allowing the agent to improve both its workflows and its strategy for developing them. We evaluate MoLRSI in two challenging molecular discovery settings that require balancing competing properties. In antibiotic discovery, MoLRSI developed a screening workflow balancing antibacterial activity, cytotoxicity and structural novelty, achieving an experimental hit rate of 15.9%, compared with 8.7% for a previous expert-led workflow using the same training data and screening library. In de novo peptide-MHC binder design, MoLRSI developed workflows that jointly consider target binding and specificity, producing eight experimentally validated hits against MART-1 and SARS-CoV-1 pMHCs with preferential binding over tested off-targets while Claude Science completely failed. These findings demonstrate how experience-guided workflow optimization can address competing discovery objectives while improving the search strategies used in subsequent campaigns.