DELBERT-2: Pretrained Fingerprint Language Models for DEL Protein Binder Prediction

AI summary

Overview

Research area: Machine learning for drug discovery — specifically self-supervised molecular representation learning and DNA-encoded library (DEL) virtual screening.

Technical level: Intermediate. Readers should be comfortable with concepts such as pretraining/fine-tuning, masked language modelling, molecular fingerprints, and train/test distribution shift, but the paper explains each in accessible terms.

Scope: The paper introduces DELBERT-2, a ModernBERT-based language model that treats sparse molecular fingerprints as text, pretrains on 2.5M unlabeled DEL molecules, and evaluates whether that pretraining improves binder prediction across six protein targets under three out-of-distribution (OOD) test protocols.

What This Paper Is About

DEL screening produces large public datasets of molecules tagged with DNA barcodes, but models trained on them often fail on compounds that differ from the training data — molecules from unseen libraries, unseen building blocks, or chemically novel regions. Because the AIRCHECK database releases only precomputed molecular fingerprints (not SMILES or structures) to protect proprietary chemistry, standard sequence- and graph-based molecular encoders cannot be used directly. The paper asks whether a transformer can be pretrained on nothing but fingerprint bits, and whether that pretraining makes hit prediction more robust when the test chemistry shifts away from the training chemistry.

Key Contributions

1. Unified pretraining on the whole DEL corpus. A single model is pretrained on approximately 2.5M molecules spanning all available AIRCHECK DEL libraries, replacing the earlier per-target approach (about 500K molecules per target), and is shown to transfer better across targets.
2. A fingerprint tokenization scheme. Each active bit of a fingerprint becomes a discrete token (fingerprint type, bit position, value), converting high-dimensional sparse binary vectors into variable-length token sequences that a transformer can process with masked language modelling.
3. Three complementary OOD evaluation protocols. Hierarchy splits probe chemical novelty, library splits test cross-library transfer, and building-block splits assess compositional generalization — isolating three distinct failure modes that a single random split would mask.
4. A quantified practical benefit. Pretraining yields an enrichment factor at top-100 of 13.28±3.72 versus 12.36±3.75 for a no-pretraining transformer (p=0.040), described as a 7.4% improvement and roughly 13× enrichment over random selection.

Main Findings

• Pretraining beats both baselines overall. Across six targets (WDR91, WDR12, SETDB1, PLCZ1, LRRK2, DCAF7), DELBERT-2 generally improves PR-AUC and NDCG@1000 over a LightGBM ensemble baseline (5 models per run) and over the identical architecture trained from scratch (No-PT).
• Gains are largest where shift is worst. In hierarchy splits, the advantage over baselines is greatest at the strictest Tanimoto cutoff (0.85), and in building-block splits it is greatest in the strictest bucket (0bb, no building blocks shared with training) — suggesting pretraining improves robustness rather than only in-distribution interpolation.
• Absolute performance still degrades with novelty. PR-AUC falls as hierarchy cutoffs tighten from 0.65 to 0.85, as the held-out library fraction rises from 0.10 to 0.30, and as building-block overlap drops from 2bb to 1bb to 0bb, for all models including DELBERT-2.
• Compositional generalization is the weak point. In the 0bb bucket, absolute PR-AUC ranges only from 0.12 to 0.55 across models and targets. Figure 6 reports relative retention (performance as a percentage of in-distribution validation) of 20% for 0bb and 34% for 1bb for DELBERT-2, versus 14% and 24% for the no-pretraining model.
• Enrichment improves at realistic screening budgets. Mean EF@100 is 13.28±3.72 for DELBERT-2, 12.36±3.75 for No-PT, and 12.69±3.11 for LightGBM, evaluated at K ∈ {10, 20, 50, 100, 200, 500, 1000}; the reported p-value is 0.040 from a paired t-test. The paper states this translates to roughly 13× more true binders than random selection in the top 100.
• Not every individual cell favors DELBERT-2. Some reported cells are better for baselines (for example, on ST1 at library split 0.10 the OOD PR-AUC is 0.887 for DELBERT-2 versus 0.970 for No-PT); the paper's claim is about consistent improvement across protocols and settings rather than uniform cell-by-cell dominance.
• Model and data scale. DELBERT-2 is a 12-layer ModernBERT encoder with hidden size 1024 (about 100M parameters), pretrained with 15% masking for 20 epochs at a learning rate of 1×10⁻⁴; the original DELBERT used about 70M parameters, hidden size 640, and 16 layers, with 500K per-target molecules for 100 epochs. Each target dataset contains on the order of 10⁵–10⁶ labeled molecules with positive class rates of 5–11%.
• NSP was excluded from main figures. The authors note NSP was left out of multi-panel figures because of consistently lower performance (PR-AUC below 0.52 versus above 0.70 for other targets), with results kept in supplementary materials.
• Baseline protocol differences explain a gap with prior work. Table 7 notes that DELBERT's full K-fold cross-validation included folds where the held-out library was chemically very distant from all training libraries, while DELBERT-2's 20% holdout is less extreme on average; DELBERT-2 baselines also use count-based fingerprints and a larger fingerprint set.

Methodology in Plain English

Turning fingerprints into sentences. AIRCHECK gives each molecule as a set of bit vectors from several fingerprint types — ECFP4/6 and FCFP4/6 (circular fingerprints), AtomPair and TopTor (topological fingerprints), and MACCS (fixed substructure keys). Rather than feeding these sparse vectors to a tree model, the authors write down every bit that is "on" as a word, labelled by which fingerprint it came from and where it sits. A molecule becomes a variable-length word sequence, so the problem looks like text.

Pretraining like a language model. A ModernBERT encoder is trained with masked language modelling — 15% of tokens are hidden and the model learns to recover them — on about 2.5M unlabeled DEL molecules for 20 epochs. No binding labels are used at this stage; the model only learns the statistical structure of fingerprint space.

Fine-tuning for each target. For a given protein, the pretrained model is adapted with LoRA (rank 8, α=16, dropout 0.1, applied to the attention Wqkv module), which freezes the original weights and trains small low-rank matrices instead. Training uses focal loss (γ=2) with AdamW and early stopping on validation PR-AUC, a choice motivated by the 5–11% positive rate.

Fair comparison. LightGBM baselines are trained on the same fingerprint features and the same splits, as an ensemble of 5 models per run using stratified K-fold partitions with early stopping on the held-out fold. The no-pretraining transformer shares DELBERT-2's architecture but skips the pretraining stage, isolating the effect of self-supervised learning.

Splitting the data three ways. Hierarchy splits cluster molecules by Tanimoto similarity at cutoffs of 0.65, 0.75, and 0.85 so test molecules become progressively less similar to training molecules. Library splits hold out entire DEL libraries (10%, 20%, or 30% of libraries) to test transfer to unseen synthesis designs. Building-block splits hold out 40% of building blocks and bucket test molecules by how many of their building blocks were seen (2bb, 1bb, 0bb). Validation sets stay in-distribution; test sets are OOD by construction. Performance is measured with PR-AUC (robust to extreme class imbalance) and NDCG@1000 (rewards putting true binders near the top), plus enrichment factors at several top-K budgets.

Why This Matters

For research: The paper argues that OOD evaluation deserves to be standard for molecular models intended for real drug discovery, and shows that a single split can overestimate generalization when it leaves leakage paths such as shared building blocks or shared libraries. It also demonstrates that self-supervised pretraining is viable in a fingerprint-only regime, where graph neural networks and SMILES transformers are unavailable.

Real-world applications:

• Prioritizing which DEL hits to resynthesize and test experimentally, where validation budgets are small and top-100 enrichment directly reduces cost.
• Screening external commercial catalogues with compounds unlike the training libraries, which the library-split protocol is designed to mimic.
• Cross-target and cross-programme transfer within a DEL dataset, since one pretrained model serves all targets instead of a separate model per target.
• Collaborative work on confidential chemistry, where only fingerprints can be shared but useful predictive models are still needed.

Industry relevance: DEL screening is described as a cost-effective route to screening billions of compounds, and many industrial and public pipelines already rely on fingerprints. A drop-in pretrained fingerprint encoder that improves early enrichment under distribution shift is directly compatible with those existing workflows without requiring structural data.

Future Directions

• Compositional generalization. The 0bb regime remains weak for all models; the authors suggest pretraining on assembled molecules alone is insufficient and that split-aware objectives or building-block-level supervision are needed.
• Multi-modal SMILES plus fingerprints. Jointly modelling SMILES strings and fingerprint tokens could combine the interpretability of fingerprints with the sequential information in SMILES.
• Auxiliary structural-key prediction. An auxiliary task predicting MACCS keys (167 binary features) would give explicit supervision on chemically meaningful substructures; the authors report preliminary experiments suggesting multi-modal pretraining may reduce OOD degradation in building-block splits.
• Larger pretraining corpora. Expanding beyond 2.5M DEL molecules to ChEMBL (2.4M bioactive compounds), PubChem (10M+ structures), and MOSES (1.9M drug-like molecules) could cover scaffolds underrepresented in DEL libraries, with 10M+ molecules suggested as a target scale.
• Prospective experimental validation. Testing whether computational OOD gains translate into real hit rates, using orthogonal binding assays and hit-to-lead optimization, is left to future work.

Target Audience

Machine learning researchers working on molecular representation learning and self-supervised pretraining; computational chemists and cheminformaticians running DEL screening or virtual screening campaigns; and practitioners in pharmaceutical and biotech discovery teams who need to decide which compounds to synthesize and assay next. It is also useful for methodologists interested in how to design OOD benchmarks that reflect realistic deployment rather than random splits. Readers without background in cheminformatics will need to pick up fingerprint terminology (ECFP, MACCS, Tanimoto similarity) as they go, since the paper assumes familiarity.

Authors’ abstract

We present DELBERT-2, a pretrained fingerprint language model that converts sparse molecular fingerprints into unified token sequences using a ModernBERT encoder trained with masked language modelling on 2.5M molecules from the AIRCHECK DEL corpus. We evaluate across six targets (WDR91, WDR12, SETDB1, PLCZ1, LRRK2, DCAF7) under three out-of-distribution (OOD) protocols: hierarchical cluster splits probe chemical novelty, library splits test cross-library transfer, and building-block splits assess compositional generalization. DELBERT-2 consistently improves PR-AUC and NDCG@1000 relative to LightGBM ensemble baselines and transformers trained from scratch, with the largest gains in stringent OOD regimes. DELBERT-2 achieves 13.28+/-3.72 enrichment factor at top-100 vs. 12.36+/-3.75 for no-pretraining (p=0.040), representing 7.4% improvement and 13x enrichment over random selection. These results demonstrate that fingerprint-centric self-supervised learning effectively improves hit prioritization under distribution shift, enabling practical DEL virtual screening. Code is available at \hyperlink{https://github.com/GenerativeDrugDiscovery/DELBERT-2}{DELBERT-2}.