AutoSciDACT: Automated Scientific Discovery through Contrastive Embedding and Hypothesis Testing

AI summary

Overview

Research area: Machine learning methodology for scientific discovery — specifically contrastive representation learning combined with statistical two-sample hypothesis testing for novelty detection across scientific domains.

Technical level: Advanced. The paper assumes familiarity with contrastive learning objectives (SimCLR, SupCon), likelihood-ratio hypothesis testing, kernel methods, and domain-specific data formats from particle physics, astronomy, and histology.

Scope: The paper presents AutoSciDACT, an end-to-end pipeline that pairs supervised contrastive embeddings with the New Physics Learning Machine (NPLM) to detect and statistically quantify anomalous signals in high-dimensional scientific datasets, demonstrated on five datasets spanning four scientific domains (with two further studies in the appendix).

What This Paper Is About

Large scientific datasets are noisy and high-dimensional, which makes genuine novelties hard to distinguish from statistical fluctuations. Existing anomaly detection methods built on dimensionality reduction can flag individual odd datapoints but usually cannot produce the statistically rigorous statements — quantified significance, p-values — that scientific discovery requires. AutoSciDACT aims to close that gap by learning compact, semantically meaningful embeddings with contrastive pre-training and then testing those embeddings against a reference distribution with a machine-learning-based two-sample test.

Key Contributions

1. An end-to-end pipeline for novelty discovery in scientific datasets that is designed to transfer across domains, structured to mirror the steps of the scientific method (data reduction, hypothesis formulation, statistical testing).

2. A principled procedure for injecting domain knowledge into contrastive dimensionality reduction — using scientific simulations, hand-labeled data, and (optionally) tailored augmentations encoded through the supervised contrastive (SupCon) positive pair sets.

3. A statistically rigorous framework for quantifying the significance of observed anomalies via NPLM, moving beyond simply flagging anomalous datapoints to reporting p-values and Z-scores relative to a background-only null hypothesis, with sensitivity to both shape and normalization deviations.

4. A realistic demonstration across four disparate scientific domains — astronomy, particle physics, histology, and image data (plus a genomics study in the appendix) — showing strong sensitivity to percent-level signal injections.

Main Findings

• Significant detection at percent-level signal fractions: NPLM flags highly statistically significant deviations (Z ≳ 3, or p ≲ 10⁻³) with signal fractions as low as 1% across the datasets studied.

• Supervised baselines bound performance: Two fully supervised baselines ("supervised" and "ideal supervised," the latter re-training the embedding with the true signal included) provide a best-case estimate, and in some cases NPLM performs near this limit.

• NPLM beats Mahalanobis in most domains: In all but the synthetic datasets, NPLM significantly outperforms the Mahalanobis baseline, because Mahalanobis is best suited to roughly normally distributed background clusters and is not sensitive to overdensities near a cluster's bulk. On the synthetic dataset, which is built from Gaussian clusters, Mahalanobis is quite effective.

• Noisy dimensions are neutralized by embeddings: In Fig. 3(a), using over 100 toy synthetic datasets per point at a fixed fraction of 0.6% with 10k background, sensitivity on raw inputs collapses as additional random variables are added, while sensitivity is preserved across all numbers of random variables using the fixed-dimensional embeddings.

• Strong results in particle physics and astronomy: For both LIGO and JetClass, NPLM approaches the supervised limit at a Z-score of 3, which the authors state rivals or exceeds anomaly detection algorithms within those domains.

• Cross-domain transfer to life sciences: The histology results align with findings in prior work that embedding spaces built with label information outperform those based solely on data augmentations; AutoSciDACT detects localized abnormalities present in only a small fraction of tissue.

• Empirical Z-score ceiling: The upper limit of the empirical Z-scores is indicated by a gray line at roughly 2.88σ, set by the fixed number of pseudo-experiments (500), so empirical numbers are not quoted beyond that point; the asymptotic NPLM approximates the large pseudo-experiment limit at large Z.

• Breakdown at very high significance: Beyond roughly 5σ, some trends break down, but at that level (p ∼ 10⁻⁷) discovery is described as extremely clear.

• A citation error is disclosed: The authors acknowledge that reference [12] in the published version was linked to an incorrect, AI-hallucinated article, an error arising from preparing a BibTeX citation using LLMs; the updated draft includes the intended reference.

Methodology in Plain English

The pipeline has two phases.

Phase one — pre-training. An encoder is trained to squeeze raw, high-dimensional data down into a small set of features (the paper fixes the embedding dimension at d = 4 for every encoder). Rather than relying on generic augmentations like cropping or blurring, the encoder uses supervised contrastive learning (SupCon): datapoints sharing a class label are pulled together and those with different labels pushed apart. The loss is a generalization of the SimCLR contrastive loss, combined with an optional cross-entropy term weighted by λ_CE ∼ 0.1–0.5 so that the classification objective stays sub-dominant. Labels come from simulations or expert-annotated databases — a resource the authors argue is not usually a bottleneck in scientific domains.

Phase two — discovery. The learned embedding is then fed to the New Physics Learning Machine, which asks a classic statistical question: are the reference sample (known backgrounds, denoted R) and the observed dataset (unknown composition, denoted D) drawn from the same distribution? NPLM learns a deviation directly from data rather than assuming a particular signal shape. It re-frames maximum likelihood as binary classification between D (labeled 1) and R (labeled 0), using a Nyström-approximated kernel method with M ∼ √(|D|+|R|) Gaussian kernels, trained by minimizing a regularized weighted binary cross-entropy. The reference sample is deliberately made substantially larger than the observed data and is reweighted so that the expected yield under the null matches the expected experimental yield — making the test sensitive to both shape and normalization changes.

Calibration. The test statistic's null distribution is estimated from 500 pseudo-experiments ("toys"); p-values are computed empirically, or asymptotically by fitting a χ² distribution when the deviation is large enough that empirical estimation would need a prohibitive number of toys. Because sensitivity depends on the kernel width, six widths are used — the 1st, 25th, 50th, 75th, and 99th percentiles of the empirical pairwise distance distribution, plus twice the 99th percentile — and their p-values are averaged. The authors note this averaging is typically less powerful than a single "optimal" kernel and acts as a kind of look-elsewhere correction.

Datasets. The pipeline is applied to a synthetic dataset of Gaussian clusters in D meaningful plus M noisy dimensions; LIGO gravitational-wave data (50 ms two-channel time series sampled at 4096 Hz, 200 measurements per channel) with a white noise burst class held out; JetClass particle physics jets with boosted Higgs-to-bottom-quark decays held out; histology tiles at 256x256 pixels (0.44 µm/pixel) with NAFLD mouse liver as the anomaly; and CIFAR-10 with class 1 held out. Signals are injected at fractions typically f_S ≲ 0.1, with 500 NPLM pseudo-experiments per injection rate.

Why This Matters

Impact on research. The paper argues it is the first end-to-end scientific pipeline for novelty discovery with a rigorous statistical foundation based on hypothesis testing. Its distinctive move is decoupling expert knowledge — captured in labels — from the analysis itself, which is what makes a single pipeline transferable across astronomy, particle physics, histology, and images. This matters because most existing contrastive anomaly detection work reports AUROC for finding individual out-of-distribution points, not statistically quantified statements about distributional deviations, which is what scientific claims require.

Real-world applications.

• Gravitational-wave astronomy: searching for hypothetical, unobserved compact-binary sources in LIGO data while suppressing instrumental glitches.
• Particle physics: detecting rare processes such as boosted Higgs boson decays to bottom quarks at the LHC without pre-specifying a signal model.
• Pathology and toxicology: flagging localized abnormal tissue such as non-alcoholic fatty liver disease, relevant to early disease detection and to guiding pathologists' judgments on toxic compounds.
• General scientific data streams: any domain with abundant labeled simulation or expert-annotated background data and a need to flag distribution-level surprises.

Industry relevance. The pipeline's focus on statistically quantified anomalies — rather than raw anomaly scores — is relevant wherever false positives carry a real cost: manufacturing quality control, medical screening, instrument monitoring, and any setting where a flagged outlier must be justified with a defensible significance level rather than a threshold on a heuristic score.

Future Directions

1. Domain shift and epistemic uncertainties. All experiments assume the reference dataset correctly resembles the background distribution of the observed data. If the reference is built from data recorded under different conditions or from simulation, it may contain domain shifts; the authors state that incorporating domain shift and estimating associated epistemic uncertainties is left for future work, though they note both are technically feasible within NPLM and the embeddings.

2. Embedding dimensionality trade-offs. The paper fixes d = 4 and notes that this limits expressivity — visible in the LIGO and CIFAR-10 results, where the "ideal supervised" benchmark falls short of the supervised one. Larger embedding dimensions are studied in Appendix B.3, but beyond a certain point (hundreds or thousands of dimensions), NPLM's sensitivity is expected to degrade substantially due to data sparsity.

3. Reducing dependence on label quality. Since AutoSciDACT relies exclusively on domain knowledge encoded in labels, its performance correlates strongly with label quality. Labeling is easy in some domains (simulations, organ labels for histological patches) but laborious or impossible in others.

4. Extending the pipeline with known method extensions. The authors state they plan to extend AutoSciDACT through known extensions of their methods, with the broader goal of automating scientific discovery to enable rapid, comprehensive, and rigorous analysis across all data.

Target Audience

This paper is most valuable to machine learning researchers working on anomaly detection and representation learning who want to move beyond AUROC-based evaluation; to physicists, astronomers, and biomedical researchers interested in statistically defensible novelty searches in large datasets; and to methodologists interested in how domain knowledge can be injected into contrastive pre-training through label structure rather than hand-crafted augmentations. Readers without a background in likelihood-ratio testing or contrastive learning will need to consult the cited references (SimCLR, SupCon, NPLM) to follow the technical details fully.

Authors’ abstract

Novelty detection in large scientific datasets faces two key challenges: the noisy and high-dimensional nature of experimental data, and the necessity of making statistically robust statements about any observed outliers. While there is a wealth of literature on anomaly detection via dimensionality reduction, most methods do not produce outputs compatible with quantifiable claims of scientific discovery. In this work we directly address these challenges, presenting the first step towards a unified pipeline for novelty detection adapted for the rigorous statistical demands of science. We introduce AutoSciDACT (Automated Scientific Discovery with Anomalous Contrastive Testing), a general-purpose pipeline for detecting novelty in scientific data. AutoSciDACT begins by creating expressive low-dimensional data representations using a contrastive pre-training, leveraging the abundance of high-quality simulated data in many scientific domains alongside expertise that can guide principled data augmentation strategies. These compact embeddings then enable an extremely sensitive machine learning-based two-sample test using the New Physics Learning Machine (NPLM) framework, which identifies and statistically quantifies deviations in observed data relative to a reference distribution (null hypothesis). We perform experiments across a range of astronomical, physical, biological, image, and synthetic datasets, demonstrating strong sensitivity to small injections of anomalous data across all domains.