PeptSol: A Translational Benchmark for Protected-Peptide Solubility in Drug Manufacturing

AI summary

Overview

Research area: Machine learning for chemistry and drug manufacturing — specifically molecular representation learning applied to peptide process chemistry (solvent selection for liquid-phase peptide synthesis).

Technical level: Intermediate. The machine-learning evaluation protocol is standard, but the reader benefits from familiarity with peptide protecting groups, solvent descriptors, and benchmark design.

Scope: PeptSol is an open benchmark of 1388 hand-curated protected-peptide solubility measurements with fixed tasks, chemistry-aware splits, and paired baselines for evaluating molecular encoders on synthesis-relevant peptide developability.

What This Paper Is About

Drug-discovery models usually optimize properties of the finished peptide, but a drug must also be manufactured, and liquid-phase peptide synthesis requires every protected intermediate to dissolve for reaction and then be recoverable for purification. Existing peptide benchmarks do not represent the protecting groups, terminal modifications, and noncanonical residues that govern this behavior, so a high score on a conventional peptide task says nothing about utility for route development. PeptSol makes this gap measurable by releasing a source-grounded dataset and an evaluation contract under which a claimed modeling gain can be traced to a change in representation.

Key Contributions

1. An open, source-grounded dataset. 1388 protected-peptide solubility observations (1284 transcribed from eight LPPS papers published 1984–1989, plus 104 from three process patents published 2018–2025), spanning 84 parent sequences, 101 fully protected structures, and 43 solvents or mixtures. The deterministic curation and resolver pipeline is released alongside it, so the corpus can be extended to new sources without changing the evaluation.

2. A fixed evaluation contract. Tasks are fixed (three-class ordinal and binary solubility), splits are chemistry-aware (peptide-GroupKFold and Bemis–Murcko scaffold), every model receives the same 268-dimensional solvent block and the same supervised head, and uncertainty is estimated with cluster-aware bootstrap intervals that match the unit withheld by the split.

3. A split-sensitivity finding. The explicit-chemistry baseline ECFP+Desc is weakest under scaffold splitting but second-best under peptide grouping, so the learned-versus-explicit comparison is settled by the choice of split rather than by the encoder alone.

4. A seven-encoder comparison against explicit-chemistry controls. Frozen encoders spanning SMILES (ChemBERTa, PeptideCLM-2, MoLFormer), SELFIES (SELFormer), molecular graphs (PepLand), and amino-acid sequences (ESM-2, PepBERT) are benchmarked against ECFP fingerprints used alone and with 101 explicit peptide descriptors.

Main Findings

• Best scores are moderate and task-dependent. Under the peptide-held-out split, the best binary macro-F1 is 0.843 (PeptideCLM-2 on the historical subset), while the harder three-class ordinal task reaches 0.614 in that same cell. PepLand leads the augmented ordinal task at 0.631.

• No foundation model reliably beats explicit chemistry. Across seven encoders, two corpora, two tasks, and two splits, none of the 56 paired bootstrap intervals against ECFP+Desc is significantly positive, and 9 are significantly negative. Point-estimate leaders do not establish an advantage.

• Split choice changes the ranking. PeptideCLM-2 remains the leading encoder under peptide grouping, whereas scaffold splitting favors different models and compresses the gap between learned and explicit features. The authors recommend the peptide-held-out split as the primary translational test because linear protected peptides often share a common scaffold.

• Binary framing inflates apparent progress. Binary prediction is easier because it does not require isolating the small, censored partially-soluble class (6.3% of labels).

• The intermediate class carries the error. In the saved out-of-fold predictions of PeptideCLM-2 on the augmented ordinal task (82.9% overall accuracy), per-class F1 is 0.767 for insoluble, 0.890 for soluble, and 0.228 for partially soluble. Only 13 of 87 partially-soluble records are recovered; 53 are called soluble and 21 insoluble.

• Errors lean toward over-predicting solubility. 93 insoluble records are called soluble against 56 in the reverse direction — the costlier direction for screening, since a false soluble call sends an intermediate into a solvent that will not dissolve it.

• Solvent identity alone is a substantial predictor. With no peptide representation, the head reaches macro-F1 0.620 on the historical binary task against a majority-class floor of 0.394. Across the four peptide-held-out cells the solvent block alone recovers between 31% and 55% of the distance from that floor to the best encoder.

• Peptide and solvent blocks are complementary. Adding the peptide block to solvent features improves over solvent-only by 0.094 (SELFormer) to 0.214 (PeptideCLM-2) averaged over the four cells, while peptide-only sits below the combination for every encoder.

• Aqueous records fail completely. All 12 aqueous and aqueous-mixture records are misclassified, though these enter only through the patent extension and the evaluation holds out peptides rather than solvent families.

Methodology in Plain English

The authors built the dataset by hand rather than by automation. Historical rows were transcribed directly from primary sources; a multimodal extraction pass proposed patent rows, but every retained row was checked one by one against the source table. Each row stays linked to its source locator and records the protected sequence, solvent, conditions, verbatim annotation, harmonized class, any numeric mg/mL value, censoring status, and curation notes. A deterministic build then resolves names, constructs structures with RDKit, validates a typed schema, and removes exact duplicates. Automated proposals are never treated as ground truth without verification.

Because the two source regimes use different label vocabularies — a four-level letter code in the historical tables and free-text appearance descriptions in the patents — each is harmonized by its own published convention. In the historical code, letters A and B both mean the compound dissolved at the 10 mg/mL assay concentration (at 25 °C and 80 °C respectively) and are recorded as lower-bound readings; C means partially soluble and D insoluble, both uncensored at 80 °C.

Models are frozen encoders whose embeddings are concatenated with a shared 268-dimensional solvent block (Hansen parameters, hydrogen-bonding counts, seven RDKit descriptors, and a 256-bit radius-two Morgan fingerprint). Two controls replace the embedding with a 2048-bit radius-two Morgan fingerprint of the protected peptide, either alone or with 101 explicit descriptors. Everything runs through the same seven-seed XGBoost ensemble, with per-fold randomized search over eight hyperparameter combinations evaluated on a rotational validation fold. Metrics are pooled out-of-fold macro-F1 with 1000-sample percentile bootstrap intervals clustered by parent peptide or scaffold, matching the split. Pairwise comparisons resample identical clusters for both models.

Why This Matters

Impact on research. The paper turns a vague translational claim into a checkable precondition. Because the solvent block, supervised head, folds, and hyperparameter-selection procedure are held fixed, a reported gain can be attributed to a change in representation rather than to a favorable seed or fold — which matters in a low-data regime where such artifacts can dominate small aggregate differences. The negative result (no consistent foundation-model advantage over explicit chemistry) is itself useful: it sets an honest baseline that future representation work must clear.

Real-world applications:

• Solvent-panel prioritization. A model calibrated on peptide-held-out data could rank a deliberately broad solvent panel and flag cases near the decision boundary, allocating scarce experimental capacity where uncertainty is consequential.
• Route development triage. Ranking which peptide–solvent combinations merit synthesis and assay before committing laboratory resources.
• Process chemistry knowledge capture. The schema preserves raw labels, censoring, and source evidence, making label and structure errors auditable rather than silently embedded in a processed feature table.
• Extension to linked manufacturing endpoints. The data contract is designed to add coupling yield, crystallization or precipitation success, purity, and recovered mass alongside solubility.

Industry relevance. Peptide therapeutics span metabolic disease, oncology, and infectious disease, and liquid-phase synthesis offers a scalable alternative to resin-bound routes. Route development today still depends on empirical solvent screens, so a reproducible way to prioritize representations before committing to new laboratory screens has direct process-chemistry value.

Future Directions

• Prospective validation. The paper explicitly frames PeptSol as a gate, not a deployment. A decisive test must separate the sources of novelty that current splits leave confounded and must pre-register predictions before measurement so failures enter the evaluation set.

• Holdouts the benchmark cannot yet make. Transfer to new solvents (no solvent-family holdout exists), across labs (92.5% of records come from one programme), across eras (splits are not temporal; a historical-to-patent holdout is needed), and to reproducibly new peptide series.

• Better coverage and richer endpoints. The highest-value additions are modern Fmoc-era series, explicit failed experiments, replicate measurements, and linked process outcomes — plus a multi-objective extension keeping solubility, coupling yield, crystallization, purity, and solvent burden as separate endpoints rather than collapsing them into one retrospective label.

• Fine-tuning and calibration. End-to-end fine-tuning could change the ranking but would answer a different question by combining representation quality with adaptation capacity. Prospective calibration and decision-efficiency endpoints would connect representation quality to reliable experimental information per synthesis and assay.

Target Audience

Machine-learning researchers working on molecular representations and benchmark design, who need a hard, chemistry-grounded testbed where foundation models do not automatically win. Process and peptide chemists interested in what computational tools can and cannot currently deliver for route development. Benchmark methodologists interested in split design, cluster-aware uncertainty, and how evaluation contracts determine what a strong result means. The paper is candid about its limits — retrospective, dominated by one historical programme, without randomized solvents, replicates, or linked process outcomes — which makes it useful reading for anyone deciding how much weight to place on a reported model score.

Authors’ abstract

Liquid-phase peptide synthesis (LPPS) offers a scalable route to peptide manufacturing, but route development still depends on empirical solvent screens because each protected intermediate must be soluble for reaction and recoverable for purification. The protecting groups, terminal modifications, and noncanonical residues that govern this behaviour are poorly represented in existing peptide property benchmarks, limiting the ability to develop and rigorously evaluate molecular representation models for LPPS-relevant prediction tasks. We introduce PeptSol, an open benchmark of 1388 hand-curated protected-peptide solubility measurements with chemistry-aware splits. The measurements span only 84 parent sequences and 101 protected structures, and 92.5% come from a single historical programme. Across seven frozen molecular encoders, the best macro-F1 is 0.843 on binary solubility for the historical subset under a peptide-held-out split; the harder three-class ordinal task reaches 0.614 in that cell. No foundation model consistently outperforms a baseline combining chemical fingerprints with explicit molecular descriptors, and model rankings vary across prediction tasks and data splits. The dataset, evaluation protocol, and baselines establish a standard for measuring progress on molecular representations for synthesis-aware peptide developability.