MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

AI summary

Overview

• Research area: Machine learning for molecular design — specifically agentic (tool-using) large language model systems, reinforcement learning, and scaffold hopping in chemical space.
• Technical level: Advanced. The paper assumes familiarity with Bemis–Murcko scaffolds, SMILES, physicochemical descriptors, GRPO-style reinforcement learning, and agent harness design.
• One-sentence scope: The paper introduces MolHop, an agent framework that reframes scaffold hopping as navigation in a hierarchy of roughly 1.6 million commercial scaffolds, combined with molecular editing and trajectory memory, and shows that training agents at 2B, 4B, and 9B parameters with agentic reinforcement learning beats frozen frontier LLMs and one-shot generative baselines on cross-scaffold molecular discovery tasks.

What This Paper Is About

Most LLM-based molecular design agents are built for molecular optimization: they make local structural edits around a reference molecule to improve target properties. The authors argue that a different and harder problem — cross-scaffold discovery — requires molecules that leave the reference scaffold family entirely while still matching multiple properties of the reference. MolHop is designed to solve that second problem by giving agents explicit scaffold-navigation actions over a pre-organized hierarchical chemical space, plus editing tools and a memory of the search trajectory.

Key Contributions

1. Hierarchical scaffold action space. The authors extract approximately 1.6 million Bemis–Murcko scaffolds from the Enamine Screening Collection and organize them into a coarse-to-fine H1–H7 hierarchy, adding explicit scaffold navigation actions so agents can perform cross-scaffold search rather than only local edits.
2. A MolHop harness for long-horizon exploration. Scaffold navigation, molecular inspection, local editing, and search control are integrated into one environment with trajectory memory, supporting multi-turn molecular design.
3. Agentic reinforcement learning for cross-scaffold discovery. Models at 2B, 4B, and 9B scales are trained with an SFT warm-start followed by GRPO using multi-objective rewards covering property matching, lower synthetic accessibility (SA) scores, and scaffold transition requirements.
4. Empirical demonstration on two task sets. The harness improves frozen Grok-4.6 and DeepSeek-Flash-V4.1, and the trained MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro while producing candidates with larger scaffold differences.

Main Findings

• The harness alone helps frozen models. Without additional training, connecting Grok-4.6 to the MolHop harness raises joint@H1 from 41.0% to 64.0% on the drug task and from 36.5% to 69.0% on the commercial task. DeepSeek-Flash-V4.1 improves from 15.5% to 52.5% (drug) and from 17.5% to 60.5% (commercial).
• Harness gains come with larger structural change. For Grok, scaffold Tanimoto median drops from 0.292 to 0.141 (drug) and from 0.317 to 0.145 (commercial); MCS median drops from 0.913 to 0.684 and from 0.833 to 0.600. DeepSeek shows similar drops (0.381 to 0.133 and 0.357 to 0.132 for Tanimoto).
• Trained MolHop-9B leads the comparison. MolHop-9B achieves joint@H1 of 55.5% on the ChEMBL (drug) test set and 73.5% on the Enamine (commercial) test set, improving by 2.0 percentage points on the drug task and 12.0 percentage points on the commercial task over the best generation model.
• MolHop candidates are structurally more distinct. MolHop-9B has scaffold Tanimoto medians of 0.123 and 0.115 and MCS medians of 0.660 and 0.568 on ChEMBL and Enamine respectively, described as lying in the upper-left region of the joint@H1 versus scaffold MCS plots.
• Performance scales with model size. MolHop-2B, MolHop-4B, and MolHop-9B reach 28.5%, 48.0%, and 55.5% joint@H1 on ChEMBL and 33.5%, 52.5%, and 73.5% on Enamine. The authors attribute this to better coordination of scaffold selection, property matching, and SA optimization rather than simply larger structural change.
• SFT is essential for tool use. Valid Episode Rate rises from 21.80%, 18.10%, and 4.60% to 91.10%, 97.80%, and 97.80% for the 2B, 4B, and 9B models. Budget Overflow Rate falls from 71.80%, 81.70%, and 95.40% to 8.10%, 2.20%, and 2.00%, and Invalid Tool Invocation Rate falls from 29.50%, 3.30%, and 0.90% to 0.60%, 0.40%, and 0.20%.
• Ablations isolate what matters. Retraining MolHop-4B after removing molecular editing, trajectory memory, or the hierarchical scaffold library all reduce final performance. Removing the scaffold library or memory causes the more substantial decreases. Notably, the variant without the scaffold library obtains higher training reward but substantially lower validation joint@H1, showing the continuous surrogate reward can rise without improving joint success.
• Agents allocate actions differently. MolHop-2B relies mostly on navigation and scaffold selection with few edits, while MolHop-4B and MolHop-9B switch dynamically between scaffold selection and editing. Frozen general LLM agents continue editing-based exploration after the initial scaffold selection, with a smaller proportion of further scaffold changes.
• The harness works beyond library coverage. On out-of-distribution targets constructed by masking the central property region of the library, DeepSeek + MolHop harness reaches 43.4% joint@H1, improving substantially over scaffold retrieval alone.

Methodology in Plain English

The authors start from a commercial compound collection (Enamine Screening Collection, which the appendix states contains 4,805,572 records). After standardization and deduplication they obtain 1,811,153 unique scaffold records, then filter out scaffolds containing bridgehead or spiro structures (145,538 excluded records: 87,329 spiro-only, 55,729 bridgehead-only, 2,480 both), leaving 1,665,615 valid scaffold leaf nodes.

Those scaffolds are sorted into seven levels of abstraction, H1 to H7, by deterministic graph rules rather than similarity clustering. H1 has 14 nodes (ring-count category), H2 has 498 (connectivity topology between rings), H3 has 1,451 (connection types between rings), H4 has 14,378 (minimal scaffold topology), H5 has 153,699 (ring sizes and reduced connecting chains), H6 has 412,853 (complete generalized connectivity graph), and H7 has 1,665,615 (exact typed Murcko scaffolds). The agent navigates this tree from coarse regions down to exact scaffolds.

The harness exposes nine external actions in four groups: scaffold navigation (nav_where, nav_list, nav_filter, nav_pick), molecular inspection (mol_show, mol_site), local editing (mol_edit, with nine subactions for fragments, atoms and bonds, and rings), and search control (mol_undo, mol_done). The agent never writes raw SMILES; it selects from environment-validated actions and receives new states, editable sites, and property feedback.

Memory is maintained by the environment, not by the model. It records the reference target, current candidate state, property feedback, evaluation status, best historical record, scaffold visit trajectory, editing history, candidate-set state, and search budget, updated whenever a tool response is generated. Summaries are rule-based (by default at most 10 segments of scaffold visitation records, with the most recent 12 steps retained per segment and omitted entries counted).

Evaluation uses 200 reference molecules per test set, drawn from approved synthetic small molecules in ChEMBL and the Enamine Screening Collection. Success requires a valid molecule, a summed relative deviation MD below 1 across five properties (logP, TPSA, molecular weight, Labute ASA, molar refractivity), an SA score lower than the reference, and a different scaffold at level H_k (reported mainly at H1). Secondary metrics are scaffold Tanimoto similarity and MCS ratio.

Training is two-stage: SFT to teach the tool-use protocol, then GRPO in the MolHop environment with the terminal continuous reward Reward = max(−Cost, −4), where Cost = MD(x, x₀) + SA(x)/SA(x₀). Models are trained from Qwen3.5 at 2B, 4B, and 9B scales. Agent-based methods use a uniform budget of at most 50 assistant turns. The comparison also includes Lee et al. (2025) as an oracle-driven method given 10,000 oracle calls.

Why This Matters

• Impact on research: The paper argues that scaffold hopping should be a first-class search action rather than an incidental byproduct of molecular optimization, and shows that structuring the chemical and action space can unlock cross-scaffold ability in frozen LLMs without any fine-tuning. It also demonstrates that agentic RL can learn to trade off exploration of new scaffold regions against property recovery.
• Real-world applications:
  • Patent circumvention, where a candidate must match the function of a reference compound while sitting outside its structural family.
  • Substitution of restricted substances with structurally different alternatives.
  • Toxicity mitigation, where removing a problematic scaffold is the point of the redesign.
  • General lead diversification in drug discovery, where matched physicochemical profiles (logP, TPSA, MW, Labute ASA, MolMR) are needed alongside a new scaffold.
• Industry relevance: The roughly 1.6 million-scaffold action space is built from a commercially available screening collection, so the search is grounded in compounds that can actually be sourced. The result that a 9B open model outperforms frontier API models on this task is relevant to organizations weighing in-house training versus relying on closed models, and the two-stage SFT-then-RL recipe is directly reusable for other tool-use chemistry environments.

Future Directions

• Extend beyond the current scaffold library. The OOD experiment masks a central property region of the library and shows the harness can still reach 43.4% joint@H1 with DeepSeek, but how far beyond library coverage the approach generalizes is left open.
• Widen the property set. Evaluation is fixed to five physicochemical properties (logP, TPSA, MW, Labute ASA, MolMR) and the SA score; whether the reward design transfers to biological activity, selectivity, or ADMET endpoints is not reported.
• Improve the weak component. In ablations, removing molecular editing produced a relatively small performance drop compared with removing memory or the scaffold library, suggesting editing contributes mainly post-selection property adjustment — an area with room for a stronger design.
• Address the surrogate-reward gap. The scaffold-library-free variant achieved higher training reward but lower validation joint@H1, so closing the mismatch between the continuous cost objective and the discrete joint success metric is an explicit open problem raised by the results.

Target Audience

Researchers and practitioners in AI-driven molecular design who care about agent architectures and reinforcement learning rather than only generative models; medicinal and computational chemists interested in scaffold replacement and lead diversification; and LLM agent researchers interested in how structured action spaces and environment-maintained memory change long-horizon search behavior. Readers need comfort with cheminformatics descriptors and RL terminology to get the most out of the results tables.

Authors’ abstract

Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules. We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory memory to support long-horizon search processes. On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules. These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling agents to explore new structural regions rather than merely refine existing molecules.