Generalization Beyond Benchmarks: Evaluating Learnable Protein-Ligand Scoring Functions on Unseen Targets

AI summary

Overview

• Research area: Machine learning for structure-based drug discovery, specifically learnable protein–ligand scoring functions and their out-of-distribution generalization to novel protein targets.
• Technical level: Intermediate. The paper assumes familiarity with docking, binding affinity prediction, graph neural networks, and self-supervised pretraining, but its central argument—that standard benchmarks overstate real-world performance—is accessible to a broad computational audience.
• Scope (1 sentence): The paper evaluates two state-of-the-art learnable scoring functions (GEMS and GenScore) plus two ATOMICA-embedding variants on deliberately strict, pocket-dissimilar train–test splits covering seven CASF-2016 target clusters, and tests whether pretrained embeddings or small amounts of target-specific data can close the resulting generalization gap.

What This Paper Is About

Learnable protein–ligand scoring functions report strong results on standard benchmarks such as CASF-2016, DUD-E, and DEKOIS2.0, but those benchmarks often contain target–ligand combinations that resemble the training data, so their scores may reflect memorization rather than genuine generalization. This paper asks how much performance actually remains when a scoring model is tested on protein pockets that are structurally distinct from everything it saw during training, and whether self-supervised ATOMICA embeddings or a small number of target-specific affinity measurements can recover some of that lost performance.

Key Contributions

1. A systematic OOD evaluation of scoring functions. The authors evaluate GEMS and GenScore, together with two ATOMICA-based variants, on train–test splits built from PLINDER pocket-level clustering (pocket_lddt__50__community clustering with lDDT). Seven CASF-2016 targets (PDB IDs 1NVQ, 1SQA, 2P15, 2VW5, 3DD0, 3F3E, 3O9I) are held out entirely, so test pockets are structurally distinct from training pockets. This targets the scoring problem directly, unlike earlier leakage analyses focused on docking and pose generation.
2. A test of large-scale self-supervised pretraining as a remedy. ATOMICA embeddings (32-dimensional graph-level vectors, generated for 19,189 complexes, or 98.7% of PDBbind) are injected into GEMS (GEMS_ATOMICA) and also used to train a simple MLP (ATOMICA-MLP), to see whether broad structural pretraining transfers to novel-target scoring.
3. An analysis of how limited target-specific data can be used. For each test target, 25 complexes are held out and used either as a fixed validation set (for early stopping) or for 25-epoch fine-tuning at a small learning rate, yielding GEMS_VAL, GEMS_ATOMICA_VAL, GEMS_FT-25, and GEMS_ATOMICA_FT-25.
4. Qualitative evidence that ATOMICA space is structured. t-SNE projections of ATOMICA embeddings (two components, perplexity 30) show gradients of experimental affinity and molecular weight, and show that the difficult 3F3E cluster concentrates in the small, well-separated cluster of heavy molecules.

Main Findings

• Standard benchmarks overstate real-world performance. GEMS and GEMS_ATOMICA reach Pearson correlations of 0.815 and 0.808 on CASF-2016 (Table 1), but only averages of 0.470 and 0.498 across the seven novel-target clusters (Table 2). The best single novel-target result is 0.736 (GEMS_ATOMICA on 1SQA). Clusters 3DD0, 2P15, 3F3E, and 3O9I yield correlations below 0.5 for every method.
• No method wins everywhere. GenScore is best on 3DD0 (0.481) and 3O9I (0.480) but collapses to a near-random 0.047 on 3F3E. The simple ATOMICA-MLP reaches top performance on 2P15 (0.479), and GEMS leads on 1NVQ (0.609).
• ATOMICA embeddings improve robustness on unseen proteins. GEMS_ATOMICA beats GEMS on every cluster except 1NVQ, and achieves both the highest average (0.498) and the highest minimum (0.311) among the four methods on the OOD splits—an advantage that the CASF benchmark does not reveal.
• ATOMICA embeddings give no benefit on the easier benchmark. On PDBbind CleanSplit evaluated on CASF-2016, GEMS_ATOMICA (0.808) is marginally worse than GEMS (0.815). ATOMICA-MLP is not competitive overall (0.583 on CASF-2016) but reaches 0.704 on the independent CASF subset.
• Training dynamics indicate overfitting under standard splits. Performance curves (Figure 3) show peak scoring power early in training followed by decline, and CASF performance stays consistently above novel-target performance as training progresses. Models trained with stratified k-fold splits tend to overfit, and target-specific validation data enables better early stopping and model selection.
• Limited target-specific data narrows the gap, but not uniformly. Validation and fine-tuning each help some clusters and hurt others. Fine-tuned models are strongest overall: GEMS_ATOMICA_FT-25 reaches the highest average correlation (0.550) and GEMS_FT-25 the highest minimum (0.386), both exceeding the no-extra-data results. Both strategies improve average and worst-case performance.
• ATOMICA's embedding space organizes ligand–pocket interactions meaningfully. The large crescent-shaped cluster in the t-SNE projection shows gradients of affinity and molecular weight, while the heaviest ligands concentrate in a smaller, well-separated cluster; the three evaluated test clusters occupy distinct regions, and the hard 3F3E cluster sits mostly in the heavy-molecule region.
• A preliminary signal of reduced docking and screening power. For GenScore on the 1NVQ cluster (n=50 targets for docking, n=10 for screening, three cross-validation folds), the paper reports a performance drop relative to the full CASF-2016 decoy sets in Supplementary Figure S3; the authors note other clusters did not provide enough samples for statistically stable results.

Methodology in Plain English

The authors start from PDBbind v.2020 (approximately 19,000 complexes) as the training source and CASF-2016 (285 high-quality complexes) as the familiar benchmark. They first reproduce the standard PDBbind CleanSplit setup, which removes near-duplicate complexes based on joint ligand, pose, and pocket similarity.

To build a harder test, they use PLINDER's pocket-level clustering to select seven CASF targets from diverse protein families. For each target, every complex in its cluster goes into a separate test set, and the remaining data is used for training and validation. CleanSplit filtering is also applied to the training and validation portions of these OOD splits, so CASF and OOD numbers can be compared directly. Models are trained with 5-fold cross-validation using label-based stratification, and the five resulting models are combined into an ensemble by averaging predictions. The authors retrain the original GEMS and GenScore implementations on the same splits using their published scripts to keep comparisons fair.

Evaluation focuses on scoring power, measured as the Pearson correlation between predicted and true affinities; affinity labels are experimental Ki, Kd, or IC50 values expressed as pK values. RMSE is also reported. In parallel, ATOMICA graph-level embeddings are generated for ligand–pocket complexes in PDBbind and visualized with t-SNE, and two additional models are built: an MLP trained directly on those embeddings, and a GEMS variant that concatenates ChemBERTa ligand embeddings with ATOMICA embeddings. Because embedding extraction failed for 1.3% of complexes (mainly ligand fragmentation), those variants use marginally smaller training and validation sets, while all test sets are kept identical across methods.

Finally, to mimic a laboratory with a handful of measurements for a protein of interest, 25 complexes per test target are set aside and used either for validation-based early stopping or for short fine-tuning of the best cross-validation model.

Why This Matters

The paper argues that reported progress in ML-based scoring may partly reflect improved memorization rather than true generalization, echoing prior findings on GEMS. It calls for stricter, more realistic benchmarks as standard practice and shows that the choice of evaluation protocol can change which model looks best—ATOMICA embeddings only reveal their advantage once the splits become hard.

Real-world applications:

• Virtual screening for understudied targets, where a pharmaceutical team must rank compounds against a protein with few or no known complexes.
• De novo designed proteins and synthetic biology, where engineered binding sites have no natural counterparts in training data and small-molecule modulators are needed for external control.
• Drug discovery for poorly represented targets such as the APOE4 variant (a cluster of only two complexes with nearly identical ligands in PDBbind) and the KEAP1 Kelch domain (18 highly similar pockets).
• Laboratory workflows with sparse proprietary data, where a small set of crystal structures and affinity measurements is all that is available and must be exploited through validation or fine-tuning.

Industry relevance: scoring functions guide both docking and virtual screening of large chemical libraries, so over-optimistic benchmark numbers can translate into costly late-stage failures. The finding that 25 additional target-specific complexes measurably improve average and worst-case performance gives a concrete, low-cost operational recommendation for teams already generating their own structural and affinity data.

Future Directions

• Adopt more restrictive data splits as routine practice and systematically re-evaluate existing scoring methods to identify what actually drives generalization rather than memorization.
• Extend the strict OOD analysis to docking and screening, motivated by the preliminary drop observed on the 1NVQ cluster, which the authors note was based on a single target.
• Quantify how performance gains scale with the amount of available target data and test more advanced fine-tuning techniques beyond 25 epochs at a small learning rate.
• Move beyond graph-level ATOMICA embeddings to block-level (residue, nucleotide, or chemical motif level) or atom-level representations for protein–ligand modeling.
• Build models that capture underlying physical principles of binding, which the authors suggest will be necessary for robustness to entirely new proteins.

Target Audience

This paper is most useful for machine learning researchers and computational chemists working on protein–ligand scoring, docking, and virtual screening; benchmark designers and method developers who need to know whether reported gains survive stricter evaluation; and industrial drug-discovery teams deciding how much to trust published scoring performance and how best to use their own small, target-specific datasets. Readers interested in self-supervised pretraining for molecular modalities will also find the ATOMICA embedding analysis relevant.

Authors’ abstract

As machine learning becomes increasingly central to molecular design, it is vital to ensure the reliability of learnable protein-ligand scoring functions on novel protein targets. While many scoring functions perform well on standard benchmarks, their ability to generalize beyond training data remains a significant challenge. In this work, we evaluate the generalization capability of state-of-the-art scoring functions on dataset splits that simulate evaluation on targets with a limited number of known structures and experimental affinity measurements. Our analysis reveals that the commonly used benchmarks do not reflect the true challenge of generalizing to novel targets. We also investigate whether large-scale self-supervised pretraining can bridge this generalization gap and we provide preliminary evidence of its potential. Furthermore, we probe the efficacy of simple methods that leverage limited test-target data to improve scoring function performance. Our findings underscore the need for more rigorous evaluation protocols and offer practical guidance for designing scoring functions with predictive power extending to novel protein targets.