Insights into the Unknown: Federated Data Diversity Analysis on Molecular Data

AI summary

Overview

Research area: Federated learning (FL) applied to molecular data analysis, sitting at the intersection of privacy-preserving machine learning, cheminformatics, and AI-driven drug discovery.

Technical level: Intermediate. Readers should be comfortable with clustering concepts, dimensionality reduction, and basic molecular representations, but the paper's framing is largely conceptual and metric-driven rather than heavily mathematical.

Scope: This paper benchmarks three federated clustering approaches against their centralized counterparts and a random baseline on eight molecular datasets, and introduces a chemistry-informed evaluation metric (SF-ICF) to judge whether federated clustering produces chemically meaningful groupings.

What This Paper Is About

AI for drug discovery is limited because models are mostly trained on public data, while the most valuable data sits privately inside pharmaceutical organizations. Federated learning can train on that private data without moving it, but it removes the global visibility needed to answer basic data-centric questions such as how diverse a combined dataset is, how to build honest train/test splits, and what the shared chemical space looks like.

The paper's goal is to determine how well federated clustering methods can disentangle and represent distributed molecular data, and to show that standard mathematical clustering metrics alone are not enough to answer that question.

Key Contributions

1. A benchmark of three federated clustering methods — Federated k-Means (Fed-kMeans), Federated PCA combined with Fed-k-Means (Fed-PCA+Fed-kMeans), and Federated Locality-Sensitive Hashing (Fed-LSH) — against their centralized counterparts (upper baselines) and a random clustering assignment (lower, noise-based baseline).
2. The introduction of a chemistry-informed evaluation metric, Scaffold-Frequency Inverse-Cluster-Frequency (SF-ICF), which combines intra-cluster purity (SF) and inter-cluster rarity (ICF) of molecular scaffolds, plus its generalization to any metadata feature group as {Feature Group}-F-ICF.
3. Evaluation on eight large and diverse molecular datasets drawn from the PharmaBench collection (6,051; 14,107; 16,822; 14,775; 25,332; 3,381; 39,665; and 900 molecules).
4. An in-depth explainability analysis combining Random Forest-based feature group importance with cluster count and size statistics, intended to resolve the ambiguity left by the quantitative metrics alone.

Main Findings

• Fed-LSH wins on geometric metrics: Fed-LSH achieves the best scores for both FL client settings on the Euclidean-based Silhouette and Davies-Bouldin (DB) scores, indicating strong intra-cluster cohesion and inter-cluster separation in the Euclidean sense.
• Fed-PCA+Fed-kMeans wins on Calinski-Harabasz: This method achieves the best CH score, showing balanced clusters with respect to the ratio of inter- and intra-cluster variance.
• Chemistry-informed results point elsewhere: On the SF-ICF metric, Fed-kMeans and Fed-PCA+Fed-kMeans achieve the best scores, indicating their clusters align well with underlying molecular scaffold structure — even though Fed-LSH looked best under the standard mathematical metrics.
• The Tanimoto-based Silhouette score failed to discriminate: All centralized and federated methods performed poorly on this metric, with values around or below 0.0 and in some cases worse than the random noise baseline, so it was excluded from further results and discussion.
• The federated-to-centralized gap was smaller than expected: Federated approaches in some cases even outperformed the centralized baselines on certain metrics. The paper notes, however, that the SF-ICF comparison is biased by the scaffold-based data partitioning, which pre-groups molecules with similar scaffolds on clients and thus gives federated methods an implicit advantage.
• All methods beat noise: Except for the excluded Tanimoto-based Silhouette score, all methods consistently outperformed the random noise clustering baseline across all metrics.
• Hyperparameter sensitivity differs by method: Fed-kMeans is highly sensitive to the choice of k; larger k improved Euclidean Silhouette and SF-ICF but caused drops on other metrics, potentially due to overclustering. Fed-PCA+Fed-kMeans differed between 5 and 10 clients only in the number of FL rounds and appeared robust to the number of clients, performing close to or better than centralized PCA+kMeans except on the CH score. Fed-LSH used N_he = 32 for both client settings, showing robustness to client count.
• Scaffolds dominate the explainability analysis: Across all three methods, the most important metadata feature group for reverse-engineered cluster labels was the molecule's scaffold. This is consistent with the high SF-ICF scores and is quantitatively supported by the generalized {Feature Group}-F-ICF scores (for example, Scaffold-F-ICF of 0.997 for Fed-kMeans, 0.927 for Fed-PCA+Fed-kMeans, and 0.875 for Fed-LSH on FL client 1 of 5 of the ames data).
• The explainability analysis flags Fed-LSH overclustering: On FL client 1 of 5 of the ames dataset, Fed-kMeans and Fed-PCA+Fed-kMeans each produced 5 clusters (mean size 237.2), while Fed-LSH produced 480 clusters with a mean size of 2.47 — below the theoretical maximum of 2^N_he, and smaller than the average 3.58 molecules per shared scaffold, which the authors interpret as evidence that Fed-LSH overclusters. Even clustering purely by shared scaffolds would produce a coarser partition.
• SF-ICF complements distribution divergence measures: Comparing per-cluster SF-ICF against scaffold-based Kullback-Leibler divergence (KLD) showed agreement at the extremes (small scaffold-focused clusters score high on both; large heterogeneous clusters score low on both), but mid-sized clusters with high SF-ICF and lower KLD, an expected consequence of SF-ICF rewarding scaffold purity and rewarding low fragmentation of scaffolds across clusters.

Methodology in Plain English

The authors started from the PharmaBench collection of eight molecular datasets and discarded its classification and regression labels, because they only cared about unsupervised structure. They enriched each molecule with two representations: Murcko scaffolds (the core ring systems and linkers left after side chains are removed) and extended-connectivity fingerprints (ECFPs; radius 2, 2048 bits) computed with RDKit. The fingerprints are the input to the clustering algorithms, and the scaffolds are the basis for the chemistry-informed scoring.

To imitate a real federation, they split each dataset across clients by scaffold, giving each client an approximately equal number of unique scaffolds and varying the client count between 5 and 10. Because real pharmaceutical companies are unlikely to hold completely disjoint sets of scaffolds, they used a "soft" assignment: roughly 90% of the molecules for a given scaffold (sampled via a Dirichlet distribution) went to a primary client, and the remaining 10% were spread randomly across the other clients. This produces scaffold overlap and realistic, same-order-of-magnitude variation in molecules per client.

The three federated methods work as follows. Fed-kMeans has clients run local mini-batch k-means from a shared set of centroids (k-means++ initialization in round one) and send back updated centroids plus cluster counts, which the server averages weighted by cluster size. Fed-PCA computes the joint covariance matrix across clients exactly, then centrally performs eigenvalue decomposition to build a projection matrix that is sent back to reduce the data from F to p dimensions; Fed-kMeans then runs on the reduced data. Fed-LSH has each client report the indices of its highest-entropy fingerprint bits, with the server taking the intersection as a consensus set so that molecules can be binned by identical values on those bits.

Evaluation used three standard mathematical metrics (Silhouette, Calinski-Harabasz, Davies-Bouldin, all with Euclidean distance via scikit-learn), a Tanimoto-based Silhouette score, and the newly proposed SF-ICF, which weights each scaffold's frequency within a cluster by how rare that scaffold is across clusters. The authors first ran a dense hyperparameter grid search, ranking configurations by averaging ranks across metrics, clients, and datasets, and then benchmarked only the optimal configurations. Finally, for explainability they trained a Random Forest to predict cluster labels from metadata and aggregated feature importances by feature group, and separately examined cluster counts and size distributions.

Why This Matters

Impact on research. The paper argues that conventional clustering metrics are insufficient for judging clustering quality in molecular chemistry, and that federated clustering should be evaluated with domain-specific metrics and local, on-client sanity checks. It takes a first step toward federated diversity analysis — structured insight into distributed molecular data without compromising privacy — and provides a foundation for more robust and trustworthy federated learning in AI-driven drug discovery. It also directly extends the line of work by Simm et al. on federated data splitting, which focused narrowly on split generation rather than on the broader question of how federated clustering represents distributed chemical structure.

Real-world applications.

• Privacy-preserving collaboration across pharmaceutical organizations that want to know what a partner's private compound collection actually contains without exchanging molecules.
• Constructing meaningful training, validation, and test splits across federated clients, so that structurally similar molecules do not appear in both training and test sets and produce overoptimistic performance estimates.
• Data valuation: assessing whether a new partner's data adds diversity or largely duplicates what is already available.
• Identifying overlapping or sparse regions of a distributed chemical space, which is non-trivial once raw pairwise similarity computation across all data is no longer possible.

Industry relevance. The work originates from Apheris AI (Berlin), with an author also affiliated with the MIC at the German Cancer Research Center and the Medical Faculty of Heidelberg University, and the framing is explicitly industrial: the gap it targets is the reliance of AI methods on public datasets that lack the scale and relevance of proprietary pharmaceutical data. The use of the PharmaBench collection, which carries curated structures and ChEMBL metadata representative of real pharmaceutical applications, reinforces the deployment-oriented intent.

Future Directions

• Expand the study to a broader range of federated clustering algorithms and adopt adaptive hyperparameter strategies.
• Broaden benchmarking to more datasets and more federated setups, including different data partitioning schemes across clients and different client counts, and add statistical analysis of the results.
• Provide a deeper mathematical analysis of the introduced SF-ICF score.
• Introduce a metric that accounts for cluster counts and sizes, so that overclustering (as seen with Fed-LSH) can be detected more directly by evaluation rather than only through post-hoc explainability.

Target Audience

Researchers and practitioners working on federated learning, privacy-preserving machine learning, or data-centric AI in the life sciences. It is especially relevant to cheminformatics and computational drug discovery teams who need to analyze or partition distributed proprietary molecular data, and to machine learning engineers building federated infrastructure who must choose a clustering approach without central access to the data. Readers interested in evaluation methodology — particularly the argument that domain-specific metrics should accompany generic mathematical ones — will also find the SF-ICF proposal and the explainability analysis useful.

Authors’ abstract

AI methods are increasingly shaping pharmaceutical drug discovery. However, their translation to industrial applications remains limited due to their reliance on public datasets, lacking scale and diversity of proprietary pharmaceutical data. Federated learning (FL) offers a promising approach to integrate private data into privacy-preserving, collaborative model training across data silos. This federated data access complicates important data-centric tasks such as estimating dataset diversity, performing informed data splits, and understanding the structure of the combined chemical space. To address this gap, we investigate how well federated clustering methods can disentangle and represent distributed molecular data. We benchmark three approaches, Federated kMeans (Fed-kMeans), Federated Principal Component Analysis combined with Fed-kMeans (Fed-PCA+Fed-kMeans), and Federated Locality-Sensitive Hashing (Fed-LSH), against their centralized counterparts on eight diverse molecular datasets. Our evaluation utilizes both, standard mathematical and a chemistry-informed evaluation metrics, SF-ICF, that we introduce in this work. The large-scale benchmarking combined with an in-depth explainability analysis shows the importance of incorporating domain knowledge through chemistry-informed metrics, and on-client explainability analyses for federated diversity analysis on molecular data.