Hash Collisions in Molecular Fingerprints: Effects on Property Prediction and Bayesian Optimization

AI summary

Overview

Research area: Machine learning for chemistry, specifically molecular representation (fingerprints), Gaussian process (GP) regression for molecular property prediction, and Bayesian optimization (BO) for molecular design.

Technical level: Intermediate. The paper assumes familiarity with molecular fingerprints, kernel methods/Gaussian processes, and Bayesian optimization, but each concept is introduced in its own background section.

Scope: A controlled empirical study of whether eliminating hash collisions in molecular fingerprints (by using exact, variable-length fingerprints instead of fixed-length hashed ones) improves GP-based property prediction and Bayesian optimization on five DOCKSTRING protein-ligand docking targets.

What This Paper Is About

Molecular fingerprints turn molecules into vectors of numbers, but the standard practice of forcing them into a fixed length (usually 1024 or 2048 dimensions) uses hash functions, so two chemically different substructures can land in the same vector position. These "collisions" make distinct molecules look more similar to each other than they actually are, which should in principle mislead any model that reasons about molecular similarity. The paper asks whether removing collisions entirely — using exact fingerprints whose length varies with the number of unique substructures — improves accuracy in Gaussian process property prediction, and whether any such improvement carries over into Bayesian optimization performance.

Key Contributions

1. A quantitative audit of hash collisions. The authors measure collision frequency and the resulting overestimation of Tanimoto similarity across 10,000 randomly sampled molecule pairs from DOCKSTRING, at four fingerprint sizes (512, 1024, 2048, 4096).
2. A systematic regression benchmark. They compare three encodings — exact, compressed (RDKit default hash), and Sort&Slice (Dablander et al., 2024) — on five DOCKSTRING docking targets using GP regression with a Tanimoto kernel, across fingerprint dimensions 512–4096 and two hyperparameter regimes (fixed and optimized), averaged over 10 random trials.
3. A custom GP implementation for variable-length inputs. Because standard GP libraries require fixed-dimensional inputs, the authors built a GP that operates directly on RDKit fingerprint objects, computing Tanimoto similarities and storing only the resulting kernel matrices, so that performance differences are attributable to the fingerprint representation alone.
4. A negative result connecting prediction to optimization. They test whether small regression gains compound into BO gains over 1,000 iterations of expected improvement, and find that they do not, providing guidance for practitioners choosing fingerprint representations.

Main Findings

• Exact fingerprints beat compressed fingerprints consistently. Across all five DOCKSTRING targets under both hyperparameter settings, exact fingerprints outperformed the best-scoring compressed configurations. With optimized hyperparameters, R² improvements ranged from 0.006 (F2) to 0.017 (KIT).
• Performance against Sort&Slice is mixed. Exact fingerprints outperformed Sort&Slice on ESR2 and KIT, matched it on F2 and PARP1, and lost on PGR, where Sort&Slice reached an R² of 0.480 versus 0.470 for exact.
• PGR is the exception in more than one way. It was the hardest task (lowest overall R² scores) and the only target where Sort&Slice beat exact fingerprints, and it achieved its best Sort&Slice performance at the smallest dimension tested (512) rather than the typical 4096.
• Statistical testing confirms the compressed-fingerprint gap. Using one-way ANOVA followed by Tukey's HSD post hoc test at α = 0.05, exact fingerprints showed statistically significant improvements over compressed fingerprints across all targets, fingerprint sizes, and hyperparameter settings. Against Sort&Slice, differences were significant on ESR2, KIT, and PGR, but not on F2 or PARP1.
• Fixed hyperparameters sharpen the result. Under fixed hyperparameters, exact fingerprints outperformed all alternatives on every target on every metric.
• Performance scales monotonically with fingerprint dimension. For both compressed and Sort&Slice fingerprints on nearly all targets and settings, R² improved as dimension increased from 512 to 4096, approaching exact-fingerprint performance. The sole exception was Sort&Slice on PGR with optimized hyperparameters.
• Collisions and similarity overestimation shrink as dimension grows. Averaged over 10,000 molecule pairs, pairwise collisions fell from 4.87 (512) to 2.43 (1024), 1.41 (2048), and 0.48 (4096). Exact Tanimoto similarity was 0.167 at every size, while compressed similarity was 0.198, 0.183, 0.176, and 0.171 respectively, giving overestimates of 0.031, 0.0152, 0.008, and 0.004.
• BO shows no clear advantage for any fingerprint type. Exact fingerprints achieved the best AUC on only one target (ESR2, 0.973 ± 0.004), while compressed and Sort&Slice methods matched or exceeded exact performance on the remaining targets (top AUCs: F2, 0.989 ± 0.004 for both compressed sizes; KIT, 0.981 ± 0.005 for Sort&Slice at 2048; PARP1, 0.978 ± 0.006 for Sort&Slice at 2048; PGR, 0.873 ± 0.02 for Sort&Slice at 2048).
• BO trajectories look nearly identical across configurations. Representative trajectories for ESR2 and PGR follow similar optimization paths despite differing regression accuracies, suggesting the small predictive gains do not meaningfully change sequential decision-making.

Methodology in Plain English

The authors start from a simple intuition: if hashing makes different molecular pieces look like the same feature, then molecules will appear more similar than they really are, and a model that leans on similarity should suffer. To test this cleanly, they chose Gaussian processes, which are kernel methods that work directly from pairwise similarity between inputs. That matters because it isolates the fingerprint question — unlike swapping in a neural network, you are not simultaneously changing the number of model parameters.

They compared three ways of encoding the same underlying information. All three use count-based ECFPs (extended connectivity fingerprints) with radius 2, meaning they count how often each circular substructure up to two bonds out appears in a molecule, and all three compute identical substructure identifiers. The difference is purely in the encoding: exact fingerprints keep every unique substructure, producing variable-length vectors; compressed fingerprints fold those into fixed vectors of 512, 1024, 2048, or 4096 dimensions using RDKit's default hash function, which causes collisions; and Sort&Slice picks the most frequently occurring substructures from a reference dataset (ZINC250k) rather than hashing, which eliminates collisions for common substructures but not all. Count-based rather than binary fingerprints were used deliberately, so the compressed version carries as much information as possible and exact fingerprints get no artificial advantage.

Since off-the-shelf GP libraries demand fixed-length inputs, the team wrote a custom GP that accepts RDKit fingerprint objects and computes Tanimoto similarities directly, storing only the kernel matrices. They benchmarked on five protein-ligand docking targets from DOCKSTRING — PARP1 (enzyme, easy), F2 (protease, easy-medium), KIT (kinase, medium), ESR2 (nuclear receptor, hard), and PGR (nuclear receptor, hard) — using the predefined cluster-based train/test split to avoid leakage from structurally similar molecules. They trained on a 10,000-molecule subset of the roughly 220,000-molecule training set and evaluated on a test set of approximately 38,000 molecules using R², MSE, and MAE, under both fixed and optimized hyperparameters, averaged over 10 random trials.

For the optimization study, they simulated Bayesian optimization on the same five targets with a GP surrogate using fixed hyperparameters. The candidate pool was the complete dataset (training plus test), the initial training set was 1,000 molecules randomly drawn from the bottom 80% of the data, and the budget was 1,000 iterations with expected improvement as the acquisition function. They compared the same three fingerprint configurations, using dimensions of 1024 and 2048 for the fixed-length ones, and measured area under the curve of the best observed value over time, averaged over 5 random trials.

Why This Matters

The paper isolates a specific, often-overlooked implementation detail — hashing in fingerprint construction — and shows it has measurable, consistent consequences for regression accuracy, while also showing that those consequences do not automatically propagate into optimization outcomes. That distinction is useful because it separates "better predictions" from "better decisions," two things the literature often conflates.

Real-world applications:

• Virtual screening and lead identification. Docking-score prediction models inform which compounds get synthesized or purchased; a consistent if modest regression improvement changes which molecules rank highly.
• Lab-in-the-loop experimental design. GPs are widely used in active-learning workflows where each experiment is expensive, from reaction optimization to protein engineering campaigns.
• Cheminformatics tooling and defaults. Fingerprint size and hashing are default settings in widely used libraries; the results give concrete guidance on when increasing dimension is enough and when a collision-free representation is worth the extra engineering.
• Benchmarking practice. The result that BO outcomes are insensitive to small accuracy differences is relevant to anyone using BO performance as a proxy for surrogate model quality.

Industry relevance: Pharmaceutical and biotech groups running docking screens and active-learning campaigns must choose fingerprint representations for every model they deploy. This work suggests that for pure prediction tasks, reducing collisions is worth modest effort, but teams should not expect that investment to translate automatically into better optimization campaigns.

Future Directions

• Investigate the link between surrogate accuracy and BO performance, including the role of uncertainty estimates, which the authors note are also a key component of successful BO procedures beyond predictive accuracy.
• Resolve the Sort&Slice calibration confound. The authors acknowledge that Sort&Slice performance may be overestimated due to potential data leakage, since it was calibrated on ZINC250k, the same data source used for training and evaluation; recalibrating on an independent reference dataset would test this.
• Use more challenging optimization landscapes. For even the hardest target (PGR), all BO procedures found top 0.1% molecules within several hundred iterations, which may have masked differences between fingerprint types.
• Extend the analysis to other fingerprint types and other kernel methods such as SVMs. The authors report preliminary results showing similar observations with different fingerprint types and call for a systematic analysis.

Target Audience

Practitioners and researchers in machine learning for chemistry who build GP-based property predictors or Bayesian optimization pipelines, especially those choosing fingerprint representations for docking-score prediction or lab-in-the-loop experimental design. It is also relevant to cheminformatics developers setting fingerprint defaults, and to methodologists interested in the often-assumed but rarely tested link between surrogate model accuracy and downstream optimization performance. Readers should be comfortable with basic kernel methods and molecular fingerprints; the paper supplies the necessary background on GPs and BO.

Authors’ abstract

Molecular fingerprinting methods use hash functions to create fixed-length vector representations of molecules. However, hash collisions cause distinct substructures to be represented with the same feature, leading to overestimates in molecular similarity calculations. We investigate whether using exact fingerprints improves accuracy compared to standard compressed fingerprints in molecular property prediction and Bayesian optimization where the underlying predictive model is a Gaussian process. We find that using exact fingerprints yields a small yet consistent improvement in predictive accuracy on five molecular property prediction benchmarks from the DOCKSTRING dataset. However, these gains did not translate to significant improvements in Bayesian optimization performance.