FINDER: Feature Inference on Noisy Datasets using Eigenspace Residuals

AI summary

Overview

Research area: Machine learning methodology for classification on "noisy" datasets, bridging stochastic analysis (Kosambi-Karhunen-Loève expansion), functional data analysis, and applied scientific classification.

Technical level: Advanced. The paper leans on Hilbert space theory, Bochner integration, Hilbert-Schmidt operators, and the singular value decomposition of compact operators, alongside applied benchmarks.

Scope: The paper introduces FINDER, a three-stage framework that constructs "stochastic features" for binary classification by mapping data into Hilbert spaces and using eigenspace residuals, and validates it on Alzheimer's Disease blood plasma proteomics and satellite-based deforestation detection.

What This Paper Is About

Standard classifiers struggle when the nominal data dimension F is much greater than the sample size N, when signal-to-noise ratios are poor, or when data collection is faulty. FINDER addresses this by treating an empirical dataset as one realization of an underlying random field (with no assumptions on its distribution), mapping it into a Hilbert space, and using the generalized Kosambi-Karhunen-Loève expansion to break that stochastic feature into computable eigen-components. The goal is to build features with in-built class separability so that classes land in distinct regions of the Hilbert space, making even noisy data amenable to simple classifiers like support vector machines.

Key Contributions

1. A generalized Kosambi-Karhunen-Loève expansion theorem (Theorem 2.1) that drops the usual Hilbert space separability assumptions and admits a Fourier-like expansion of stochastic features into eigen-pairs {λ_r, φ_r} plus zero-expectation random coefficients {Y_r}.

2. A set of supporting results making the framework computable, including an isometric isomorphism between L²(Ω, ℋ) and the space of Hilbert-Schmidt operators HS(ℋ, L²(Ω)) (Lemma 2.1), an optimal-truncation result identifying the best M-dimensional subspace as Span{φ_r}_{r=1}^M (Lemma 2.2), and distribution-agnostic concentration bounds via Markov's inequality (Lemma 2.3).

3. A family of residual eigenspace construction algorithms, namely Direct Residual Subspaces (extending Lakhina et al. 2004 to the KLE setting), Multi-Level Subspaces (MLS, adapting Tausch and White 2003), and the novel Anomalous Class Adapted methods ACA-S and ACA-L presented in Algorithm 1.

4. State-of-the-art empirical results in two data-deficient scientific domains: Alzheimer's Disease stage classification from blood plasma protein data and remote sensing detection of deforestation using Sentinel-2 optical and Sentinel-1 SAR data, with a public codebase at Murphy et al. (2025).

Main Findings

• ADNI proteomics AUC gains: On 146 blood plasma biomarkers, FINDER variants reached maximum AUCs of 0.894 (MLS, RBF), 0.970 (ACA-S, linear), and 0.937 (MLS, RBF) on the AD vs. CN, CN vs. LMCI, and balanced-regime cohorts respectively, against a best benchmark of 0.789, 0.910, and 0.789. In the unbalanced regime, AD vs. CN reached 0.913 (ACA-L, RBF) versus a 0.789 benchmark.

• Large runtime reductions: FINDER's feature transformation plus classifier training took 70.6 ms to 287.0 ms per LPOCV round depending on cohort and variant, compared to 246.69 ms to 2302.33 ms for the best benchmark methods.

• Accuracy improvements beyond AUC: FINDER elevated SVM accuracy over benchmark learners on all three ADNI cohorts (Tables 4 and 5 in the paper's appendix). As an illustration of why AUC alone is insufficient, an SVM with a linear separating hypersurface on the CN vs. LMCI cohort (N_A = 346, N_B = 54) obtained an AUC of 0.91 but an accuracy of only 0.71.

• Sensitivity to the residual dimension: Performance is sensitive to the choice of M_res; the overall trend is that larger M_res achieves higher AUCs, though too large M_res can diminish them. MLS remained robust to pre-balancing the data, while ACA was highly dependent on that choice.

• Deforestation detection with less optical data: Under postprocessing, HMM + FINDER (Hybrid) achieved 0.942 overall accuracy with 71 training days and 0.933 with only 35 training days. FNRT (Hybrid) achieved 0.935 overall accuracy only with 130 training days, and dropped to 0.260 with 71 days. FINDER matched FNRT's best accuracy while using roughly 45 percent less data when FNRT is usable, and retained performance where FNRT is not applicable.

• Robustness to loss of optical imagery: Simulating all n ≤ 161 Sentinel-2 EVI days (100 random sets per value), optical-only accuracy degraded significantly as the number of optical days decreased, while the hybrid method remained robust, bottoming out at the SAR-only result.

• HMM + FINDER (Optical) alone achieved 0.936 overall accuracy with 71 training days, matching the joint optical + SAR HMM + FINDER accuracy (0.942), with user accuracy of 0.801 versus 0.865 and producer's accuracy of 0.748 versus 0.752.

• Comparative advantage in the literature: On the ADNI data, FINDER's results were significantly higher than those presented in Figure 2 of Rehman et al. (2024), even though that work combined a subset of the same ADNI blood plasma proteins with age, sex, education, and the APOE4 gene.

Methodology in Plain English

FINDER is a three-step framework: dataset acquisition, feature construction, and classification. The core idea is to stop treating a dataset as a fixed table of numbers and instead treat it as one draw from a random process. The data is passed through a map into a Hilbert space (often ℝ^F or L²([a,b])), producing what the authors call a stochastic feature v. Standard classifiers are then just maps from that Hilbert space to {0,1}, separated by a threshold.

Because building these stochastic features directly can be prohibitively expensive, the authors use the Kosambi-Karhunen-Loève expansion as a shortcut. It behaves like a generalized singular value decomposition, splitting v into an expectation plus a sum of eigen-components {λ_r, φ_r} weighted by random coefficients {Y_r}. The eigenvalues λ_r are the variances of the projections of v onto each φ_r, so they describe where the data concentrates and how much it spreads. A key practical move is that the Y_r have expectation zero, so they can be dropped from the probability bounds entirely, avoiding any need to estimate them or assume a distribution.

Classification then exploits separation: the dataset is centered on class A, so the eigenbasis of class A captures what A "looks like." The residual subspace H_res is chosen inside the orthogonal complement of class A's span, and class B is expected to present a different spectral profile there. Three linear ways of tiling that residual subspace are implemented: Direct Residual Subspaces (the whole orthogonal complement), Multi-Level Subspaces (a basis built adaptively), and ACA-S / ACA-L (two successive projections, minimizing or maximizing the norm of the projected class B deviation).

Experiments use leave-(one)-pair-out cross-validation on standardized data in both balanced and unbalanced training regimes, with a linear SVM or an RBF SVM as the downstream classifier. Deforestation experiments use a Hidden Markov Model instead, classifying each pixel as forest, deforestation, or cloud cover after SAR filtering and optical residual analysis are fused.

Why This Matters

Impact on research: The paper offers a distribution-agnostic theoretical route to classification under noise, showing that probabilistic concentration bounds on class separability can be derived without estimating the random coefficients of the expansion. It connects classical functional data analysis and spectral operator theory to modern classification practice, and shows that reducing noise via eigenspace projection can make nominally intractable datasets tractable for elementary ML methods.

Real-world applications:

• Alzheimer's Disease staging: Distinguishing Cognitive Normal, Mild Cognitive Impairment, and Alzheimer's Disease subjects using minimally invasive blood plasma tests, where early detection improves prognosis.

• Deforestation and illegal logging detection: Near-real-time tracking of forest loss in cloudy regions such as the West African coast, Madagascar, parts of the Amazon, and Southeast Asia, where optical satellites are frequently obstructed.

• Carbon accounting: Quantifying loss of carbon absorption in the atmosphere in regions where cloud cover would otherwise delay or block monitoring.

• Noisy industrial or scientific classification pipelines: Any domain with low signal-to-noise ratios, small sample sizes, or faulty data collection where deep learning is known to be susceptible.

Industry relevance: The method substantially lowers compute time relative to benchmark learners (for example, 70.6 ms versus 1173.42 ms in one ADNI comparison) and lets simpler algorithms such as SVMs or HMMs achieve high accuracy, offering a pathway to manageable computational costs on nominally intractable datasets. It is also relatively robust to unbalanced data, which is common in real collection efforts.

Future Directions

• Better truncation parameter selection: FINDER depends on judicious choices of M_A and M_res. Currently these rely on heuristics informed by Scree plots and user intuition, and the ACA variants show no obvious pattern as M_res varies. The authors state they are researching better methods for making these choices.

• Tighter probability bounds: The Lemma 2.3 Markov bounds are described as sub-optimal because the implementations eliminate the need to estimate Y_r. Modeling Y_r more carefully could yield more useful bounds at additional computational cost.

• Handling spectrally similar classes: If the eigen-pairs (λ_r^A, φ_r^A) and (λ_r^B, φ_r^B) are too similar, classification becomes nearly impossible. Finding remedies for this failure mode remains open.

• Extension to multi-class problems: FINDER is currently a binary classification regime, and the paper offers no breakthroughs on multi-class problems beyond decomposing them into a collection of costly binary ones.

Target Audience

This paper is best suited to researchers and graduate students in machine learning theory, functional data analysis, and applied stochastic analysis who work on high-dimensional, low-sample-size, or low-SNR classification problems. It also speaks to domain scientists in neuroimaging and proteomics, and in remote sensing and environmental monitoring, who need classification methods that remain accurate when data is scarce or clouded. Practitioners interested in pairing strong feature construction with lightweight downstream classifiers such as SVM or HMM will find the computational efficiency arguments and the released codebase useful.

Authors’ abstract

''Noisy'' datasets (regimes with low signal to noise ratios, small sample sizes, faulty data collection, etc) remain a key research frontier for classification methods with both theoretical and practical implications. We introduce FINDER, a rigorous framework for analyzing generic classification problems, with tailored algorithms for noisy datasets. FINDER incorporates fundamental stochastic analysis ideas into the feature learning and inference stages to optimally account for the randomness inherent to all empirical datasets. We construct ''stochastic features'' by first viewing empirical datasets as realizations from an underlying random field (without assumptions on its exact distribution) and then mapping them to appropriate Hilbert spaces. The Kosambi-Karhunen-Loéve expansion (KLE) breaks these stochastic features into computable irreducible components, which allow classification over noisy datasets via an eigen-decomposition: data from different classes resides in distinct regions, identified by analyzing the spectrum of the associated operators. We validate FINDER on several challenging, data-deficient scientific domains, producing state of the art breakthroughs in: (i) Alzheimer's Disease stage classification, (ii) Remote sensing detection of deforestation. We end with a discussion on when FINDER is expected to outperform existing methods, its failure modes, and other limitations.