Information-Theoretic Quality Metric of Low-Dimensional Embeddings

AI summary

Overview

• Research area: Machine learning / manifold learning — specifically the evaluation of dimensionality-reduction quality using information theory.
• Technical level: Advanced. The paper assumes familiarity with singular value decomposition, Shannon entropy, Procrustes analysis, and standard manifold-learning methods (LLE, Isomap, HLLE, KPCA).
• Scope: The paper introduces a new entropy-based quality metric for low-dimensional embeddings and compares it against established distance-based and geometric metrics on an S-curve manifold and an S&P 500 financial time series.

What This Paper Is About

Existing metrics for judging how good a low-dimensional embedding is — stress, Trustworthiness, Mean Relative Rank Error (MRRE), Local Procrustes — measure how distances, ranks, or local shapes change under projection. None of them directly measures how much information is preserved. The authors propose the Entropy Rank Preservation Measure (ERPM), which uses the Shannon entropy of the singular-value spectrum of each neighborhood matrix (expressed through the stable rank) to quantify how uncertainty changes between the original high-dimensional representation and its reduced projection.

Key Contributions

1. The ERPM metric itself. A local, per-neighborhood measure of information change, defined as ΔH_i = H(Ȳ_i) − H(X̄_i), where H is the entropy of the normalized squared singular-value spectrum of the centered neighborhood matrices, written explicitly in terms of the stable rank as ΔH_i = log(r(Ȳ_i)/r(X̄_i)) + ε(X̄_i) − ε(Ȳ_i).
2. A global summary statistic. R_ΔH, the average of ΔH_i over all n points, plus the argument that the distribution of ΔH_i matters as much as its mean.
3. An entropy–stable-rank connection. The paper shows that entropy of a matrix can be written as H(M) = log r(M) − ε(M), linking Shannon entropy directly to the stable rank r(M) = ‖M‖_F²/‖M‖² = Σσ_j(M)²/σ_1(M)².
4. A comparative empirical study. ERPM is benchmarked against MRRE (distance-based) and Local Procrustes (geometric) across five reduction methods (LLE, HLLE, Isomap, KPCA with quadratic kernel, and PCA as a linear baseline) on two datasets.

Main Findings

• Distance metrics are nearly uncorrelated with geometry and information metrics. Both MRRE components W_n and W_v exhibit extremely low correlation coefficients with Local Procrustes (R_C) and with R_ΔH. Preserving distances does not imply preserving geometry or information content.
• Local Procrustes and ERPM are strongly correlated on average. The correlation between R_C and R_ΔH is extremely high in absolute terms (the negative sign comes from their respective value ranges).
• ERPM penalizes more severely than Local Procrustes. ERPM tends to take more extreme values, while the overall trend profiles of the two metrics stay aligned for all methods.
• Behavior as a function of neighborhood size k. For the S-curve, both metrics drop sharply at small k (local instability), stabilize as k grows, and global performance keeps deteriorating for all methods except Isomap.
• The financial data is harder. Compared with the S-curve, the S&P 500 embedding produces notably more negative entropy values and substantially higher Procrustes values. PCA and KPCA2 show the least negative R_ΔH and smallest R_C, while LLE, Isomap, and HLLE show more severe losses or distortions.
• Local-level relationship is not one-to-one. For intermediate Procrustes values between 0.3 and 0.5, ΔH_i ranges from roughly −0.2 to about −0.9, meaning similar geometric fits can hide very different information losses.
• Method rankings differ by metric. In the S-curve ERPM densities, HLLE sits furthest left (most entropy loss), followed by PCA, KPCA2, and LLE shifting toward less negative values, while Isomap is sharply concentrated near zero. The Procrustes densities are much more overlapped — HLLE, PCA, and KPCA2 look very similar, so methods that differ substantially under ERPM can appear almost equivalent under Procrustes.
• Pathological neighborhoods hide behind good averages. In the financial data, methods from PCA and KPCA2 show less negative values with elongated left tails, indicating neighborhoods with very severe information loss despite acceptable global behavior.

Methodology in Plain English

The authors start from a high-dimensional data matrix and its reduced version. For each data point, they take its k nearest neighbors in the original space and in the embedding, center both neighborhood matrices, and compute the singular values of each. Squaring and normalizing those singular values turns them into a probability distribution describing how variance is spread across spectral directions; the Shannon entropy of that distribution measures how much uncertainty the neighborhood carries. The ERPM is simply the difference in that entropy before and after reduction. A single global number, R_ΔH, averages the differences across all points.

They validate this by comparing it to two established metrics. MRRE builds a co-ranking matrix summarizing how distance ranks change and counts hard intrusions, hard extrusions, and mild intrusions/extrusions. Local Procrustes aligns each low-dimensional neighborhood to its high-dimensional counterpart via SVD and measures the residual geometric error.

Two datasets were used: an S-curve with 2,000 points, and daily S&P 500 closing prices from January 1, 2000 to January 1, 2025, converted into a Takens delay embedding with an estimated optimal delay of 57 and embedding dimension of 6 (using the first minimum of auto mutual information and Cao's method, following Huang et al.). Five reduction methods were applied. Neighborhood size k was swept over [1, 20], with method-dependent values set to k_max for a consistent comparison, and the final local analysis fixed k = 15 based on the onset of stabilization.

Why This Matters

Impact on research: The paper argues that information preservation and geometric fidelity are complementary, not interchangeable, properties of an embedding. It gives manifold-learning research a metric that directly targets spectral information loss rather than inferring it from distance distortion, and it argues that reporting only global averages of quality metrics can conceal pathological local behavior.

Real-world applications (as framed in the paper):

• Building early-warning indicators for regime shifts and financial crashes, where embedding quality supports consequential decisions.
• Reconstructing attractors from time series embeddings via Takens' theorem.
• Clustering and Hidden Markov model pipelines that detect pre-crisis market dynamics from a reduced representation.
• Preprocessing for any information-sensitive task where losing spectral structure is more costly than losing distance fidelity.

Industry relevance: For finance and risk analytics specifically, the S&P 500 experiment shows that methods with nearly indistinguishable Procrustes scores can have very different entropy-loss profiles, which matters when choosing a dimensionality-reduction method for a monitoring or alerting system. The elongated left tails in the financial ERPM densities identify which methods produce neighborhoods with catastrophic information loss — information that mean-based reporting would hide.

Future Directions

• Extending the metric beyond neighborhoods defined by k nearest neighbors, and studying sensitivity to the choice of k more systematically beyond the [1, 20] sweep used here.
• Applying ERPM to more datasets and reduction methods, since the current evidence comes from one synthetic manifold (S-curve, 2,000 points) and one financial series.
• Developing a formal notion of "stable projection" analogous to the stable-embedding condition of Eftekhari et al., which the authors raise as a motivation but do not resolve.
• Using ERPM diagnostically to identify and avoid pathological neighborhoods during embedding construction, rather than only evaluating finished embeddings, and connecting the metric to the curvature-based resilience indicator discussed in the motivating case study.

Target Audience

Researchers and graduate students in manifold learning and dimensionality reduction who need to evaluate embedding quality; practitioners in quantitative finance, dynamical-systems analysis, and signal processing who use dimensionality reduction as a preprocessing step for sensitive downstream tasks; and methodologists interested in information-theoretic evaluation criteria for representation learning. Readers without a linear algebra and information theory background will find the formal sections (stable rank, entropy decomposition, Procrustes derivations) demanding, though the conceptual argument — that geometric fidelity and information preservation are different things — is accessible on its own.

Authors’ abstract

In this work we study the quality of low-dimensional embeddings from an explicitly information-theoretic perspective. We begin by noting that classical evaluation metrics such as stress, rank-based neighborhood criteria, or Local Procrustes quantify distortions in distances or in local geometries, but do not directly assess how much information is preserved when projecting high-dimensional data onto a lower-dimensional space. To address this limitation, we introduce the Entropy Rank Preservation Measure (ERPM), a local metric based on the Shannon entropy of the singular-value spectrum of neighborhood matrices and on the stable rank, which quantifies changes in uncertainty between the original representation and its reduced projection, providing neighborhood-level indicators and a global summary statistic. To validate the results of the metric, we compare its outcomes with the Mean Relative Rank Error (MRRE), which is distance-based, and with Local Procrustes, which is based on geometric properties, using a financial time series and a manifold commonly studied in the literature. We observe that distance-based criteria exhibit very low correlation with geometric and spectral measures, while ERPM and Local Procrustes show strong average correlation but display significant discrepancies in local regimes, leading to the conclusion that ERPM complements existing metrics by identifying neighborhoods with severe information loss, thereby enabling a more comprehensive assessment of embeddings, particularly in information-sensitive applications such as the construction of early-warning indicators.