Intrinsic Dimension Estimation for Radio Galaxy Zoo using Diffusion Models

AI summary

Overview

Research area: Machine learning applied to radio astronomy — specifically, intrinsic dimension (iD) estimation of radio galaxy image data using score-based diffusion models, with cross-analysis against Bayesian neural network uncertainty/energy scores.

Technical level: Advanced. The paper assumes familiarity with score-based generative models, stochastic differential equations, singular value decomposition, manifold hypothesis, Bayesian neural networks, and Hamiltonian Monte Carlo.

Scope: The paper estimates the intrinsic dimension of the Radio Galaxy Zoo (RGZ) Data Release 1 dataset using a diffusion model, compares those estimates to classical estimators (MLE, Local PCA, PPCA), and relates iD to BNN energy scores, Fanaroff-Riley morphology, and signal-to-noise ratio.

What This Paper Is About

Under the manifold hypothesis, high-dimensional data such as radio galaxy images may actually lie on a lower-dimensional submanifold whose dimensionality — the intrinsic dimension — reflects the effective degrees of freedom in the data. This paper uses a trained score-based diffusion model to estimate the iD of radio galaxy images from the Radio Galaxy Zoo Data Release 1, and asks whether that iD correlates with how "in or out of distribution" a source is according to a Bayesian neural network trained on the labelled MiraBest subset. The goal is to determine whether iD is a useful quantitative descriptor of radio galaxy data, and to compare diffusion-based estimates against traditional statistical estimators.

Key Contributions

1. Diffusion-based iD estimation on radio astronomy data: The first application (to the authors' knowledge, as presented in this work) of the score-vector-based iD estimation method of [27] to the Radio Galaxy Zoo Data Release 1 dataset, using a score-based diffusion model trained with weighted denoising score matching on a DDPM architecture.

2. Linkage of iD to BNN energy scores: The paper connects iD estimates to scalar energy scores from an HMC-based Bayesian neural network trained on MiraBest, partitioning RGZ sources into 8 mean energy intervals and 6 standard deviation intervals, and showing that out-of-distribution sources have higher estimated iD.

3. Benchmarking against classical estimators: Direct comparison of diffusion-based iD with MLE (at two neighbour settings, m = 5 and m = 20), Local PCA via the Fukunaga-Olsen method (alpha = 0.05), and Probabilistic PCA (global, per-label estimates) across the mean and standard deviation intervals.

4. Morphology and noise analysis: An examination of iD as a function of Fanaroff-Riley class (FR I vs FR II) and signal-to-noise ratio, including a comparison against iD values previously reported for natural image datasets.

Main Findings

• iD increases with out-of-distribution-ness: Estimated iD increases with interval number for both the mean and standard deviation intervals of the BNN energy distribution, indicating that objects identified by the BNN as out-of-distribution have a higher iD than in-distribution objects.

• Diffusion iD estimates exceed classical estimates: Classical estimators broadly follow the same trend as the diffusion method but produce much smaller magnitudes. In the mean-interval table, MLE (m = 5) ranges from 9.249 (interval 1) to 40.034 (interval 8), and MLE (m = 20) from 8.086 to 38.033. Local PCA ranges from 10.702 to 17.889. PPCA gives only two reportable values, 4357 and 3638, with the remaining intervals marked N/A due to insufficient samples.

• Standard deviation interval results: For intervals 1 through 6, MLE (m = 5) gives 9.318, 9.172, 17.546, 19.157, 18.394, 20.681; MLE (m = 20) gives 8.182, 7.954, 15.667, 17.207, 18.277, 20.353; Local PCA gives 10.571, 11.892, 16.907, 19.144, 20.586, 21.556; PPCA gives 4977 and 3777 for intervals 1 and 2, with N/A for intervals 3–6.

• RGZ iD exceeds natural image datasets: The diffusion iD estimates for RGZ are much larger than those reported for natural image datasets in previous work, which the authors suggest may be due to the inherently noisy nature of radio astronomy data.

• No FR I / FR II difference in iD: No relationship is found between the FR I and FR II morphological classes in terms of intrinsic dimension.

• Weak SNR trend: Overall there is no strong relationship between SNR and iD. At low iD values (below 100), FR II sources tend to exhibit higher SNRs, suggesting galaxies with lower iD are less affected by noise; however, a high SNR does not necessarily imply a low iD.

• Numerical iD values from the diffusion model are not reported in the text: The diffusion-based estimates appear in Figures 2 and 5 (and the accompanying tables list only the classical benchmarks), so the exact diffusion iD magnitudes are presented graphically rather than as numbers in the paper text.

Methodology in Plain English

The researchers trained a diffusion model — a generative model that learns to reverse a gradual noising process — on 72x72 pixel images of radio sources from Radio Galaxy Zoo Data Release 1. Diffusion models learn the "score," which is the direction in which data density increases at any point in image space. A recent theoretical result shows that near the data manifold and at small noise levels, score vectors point almost entirely toward the manifold rather than along it.

To estimate iD, the team took a data point, added noise to it 328 times (k = 328), and computed the score vector for each noisy version with the trained model. Stacking these 328 vectors as columns of a matrix, they applied singular value decomposition. The largest gap between consecutive singular values marks where the singular values collapse, giving the dimension of the "normal" space; subtracting that from the ambient dimension (72x72) yields the intrinsic dimension.

Separately, a Hamiltonian Monte Carlo Bayesian neural network trained on the MiraBest dataset (1256 images of 150x150 pixels; the Confident subset with binary FRI/FRII labels; split into 584 training, 145 validation, and 104 test samples) produced energy scores for each RGZ source using 200 posterior samples with a temperature of 1. Sources were then bucketed into 8 mean intervals and 6 standard deviation intervals based on log-normal fits to the energy distributions. The iD was estimated for each interval and compared against classical estimators and against FR labels from prior foundation-model-based classification work.

Why This Matters

Interpreting the intrinsic dimension of a dataset gives a quantitative handle on how complex the data really is, and connecting it to a model's uncertainty signal gives a way to detect when new data falls outside what a model has learned. Because labelling radio galaxies is expensive and the volume of survey data is growing quickly, methods that characterise unlabelled data without labels are directly useful.

Real-world applications include:

• Radio survey pipelines: Automatically flagging radio sources that fall outside the domain of an existing labelled catalogue, so that human annotation effort is directed where it is needed.
• Data compression and storage planning: Knowing the effective dimensionality of a dataset informs how much compression is theoretically achievable for subsets of that data.
• Model selection and representation quality: Using iD to quantitatively evaluate and improve self-supervised learning representations for astronomy.
• Noise-aware analysis: Understanding where SNR confounds or does not confound downstream dimensional analysis of survey images.

Industry relevance: The techniques here — intrinsic dimension estimation via diffusion scores, energy-based out-of-distribution detection with Bayesian neural networks, and unsupervised characterisation of large imagery datasets — transfer to any domain with large unlabelled image collections and scarce expert labels, including remote sensing, medical imaging, and industrial inspection.

Future Directions

1. Self-supervised representation analysis: Extending the approach to estimate iD from the features learned by self-supervised learning models trained on RGZ, using the results to determine what degree of compression is achievable for different subsets of the data.

2. Explaining the elevated RGZ iD: The diffusion iD for RGZ far exceeds values reported for natural image datasets; establishing whether this is driven by the noisy nature of radio astronomy data, by the survey's morphological diversity, or by properties of the diffusion training remains open.

3. Resolving the FR I / FR II question: Because no relationship was found between FR morphological classes and iD, further work is needed to determine whether iD is simply insensitive to this classification or whether the available label subsets were too limited.

4. Quantitative use of the iD–energy relationship: The authors suggest that future work could use the relationship between iD and energy scores to quantitatively study and improve the representations learned by various self-supervised learning algorithms.

Target Audience

This paper is most valuable to machine learning researchers working on generative models, manifold learning, and intrinsic dimension estimation, particularly those interested in out-of-distribution detection and representation analysis. It is also relevant to radio astronomers and astrophysicists building classification pipelines for large surveys such as RGZ, FIRST, ATLAS, and MiraBest, and to practitioners applying Bayesian deep learning to scientific imaging where labelled data is scarce. Readers without background in diffusion models or the manifold hypothesis will find the theoretical sections challenging, but the results and conclusions are accessible at a conceptual level.

Authors’ abstract

In this work, we estimate the intrinsic dimension (iD) of the Radio Galaxy Zoo (RGZ) dataset using a score-based diffusion model. We examine how the iD estimates vary as a function of Bayesian neural network (BNN) energy scores, which measure how similar the radio sources are to the MiraBest subset of the RGZ dataset. We find that out-of-distribution sources exhibit higher iD values, and that the overall iD for RGZ exceeds those typically reported for natural image datasets. Furthermore, we analyse how iD varies across Fanaroff-Riley (FR) morphological classes and as a function of the signal-to-noise ratio (SNR). While no relationship is found between FR I and FR II classes, a weak trend toward higher SNR at lower iD. Future work using the RGZ dataset could make use of the relationship between iD and energy scores to quantitatively study and improve the representations learned by various self-supervised learning algorithms.