Predict Training Data Quality via Its Geometry in Metric Space

AI summary

Overview

• Research area: Machine learning data quality, topological data analysis (TDA), and training-data diversity measurement, with experiments on text classification via BERT fine-tuning.
• Technical level: Intermediate. The paper is readable without a topology background, but it assumes comfort with distance matrices, entropy-style diversity scores, and fine-tuning terminology.
• Scope: The paper proposes diversity measures built from persistent homology (PH) computed on a dataset's pairwise distance matrix, proves they satisfy standard diversity axioms, and tests whether they predict model accuracy better than the entropy-based Vendi Score.

What This Paper Is About

Practitioners know that data quality drives model performance, but there is little principled guidance on what geometric property of a dataset makes it "good." The authors argue that both rich representation and freedom from redundancy matter, and that persistent homology can quantify these properties in a way entropy-based scores like the Vendi Score cannot. They then check whether these PH-derived numbers actually track the accuracy of models trained on the data.

Key Contributions

1. The authors show that persistent homology, used as a diversity measure, captures richer structural information than conventional distribution- or entropy-based metrics.
2. They develop multiple PH-based diversity measures — persistence entropy and PH-based Hill numbers — that quantify data quality and connect it to model performance, highlighting higher-order geometric features such as H₁ (loops) as key to capturing meaningful structural patterns.
3. They prove that the PH-based measure satisfies the axiomatic definition of diversity: effective size, the twin property, multi-scale behavior, and symmetry (formal proofs in Appendix A).
4. They provide practical guidance for dataset augmentation and data point selection, showing through a transfer-learning classification study that geometry-aware measures predict accuracy while the Vendi Score does not.

Main Findings

• PH diversity tracks accuracy, entropy does not: PH-based diversity measures are positively correlated with model accuracy for both H₀ (connected components) and H₁ (loops), whereas the Vendi Score shows a negative trend, with higher values associated with lower accuracy. The authors conclude geometry-aware indicators are reliable predictors of data quality while distributional, entropy-based measures are not.
• Diversity also stabilizes training: A negative relationship is reported between minimum H₀ values and the standard deviation of model accuracy, suggesting greater geometric diversity both boosts accuracy and reduces training variability.
• The "closest" subset is redundant and unstable: It has extremely high H₀-based Hill numbers (PEH₀¹: 489, PEH₀²⁰: 347), high H₁-based Hill numbers with short lifetimes (PEH₁¹: 376, PEH₁²⁰: 126), H₀ min of 0.0215, H₁ mean of 0.0025, and a Vendi Score of 1.143. Accuracy is 0.836 ± 0.021 — slightly better on average than the farthest subset but with the largest variance.
• The "farthest" subset is sparse and fragile: It has low H₀- and H₁-based Hill numbers (PEH₀¹: 478, PEH₀²⁰: 244; PEH₁¹: 291, PEH₁²⁰: 86), H₀ min of 0.0191, H₁ mean of 0.0029, and a Vendi Score of 1.160. Accuracy is the lowest at 0.832 ± 0.014.
• The "random" subset is the most balanced and best performing: Moderate H₀ Hill numbers (PEH₀¹: 485, PEH₀²⁰: 287; H₀ min: 0.0234) correspond to well-separated, non-redundant clusters, and moderate H₁ Hill numbers (PEH₁¹: 331, PEH₁²⁰: 123; H₁ mean: 0.0028) reflect stable loops. It achieves the best accuracy, 0.845 ± 0.013, with the lowest standard deviation, and a Vendi Score of 1.151.
• More data is not always better: The authors report that a relatively smaller training dataset (6%–19% of the original dataset) can achieve 91%–98.6% of the accuracy reported using the full dataset during fine-tuning, citing prior work.
• The undesirable extremes: A high-quality dataset should avoid both redundancy (too many overlapping data points) and sparsity (scattered, fragile structures), targeting high H₀ minimum values for cluster separation combined with moderate H₁ mean lifetimes.

Methodology in Plain English

The researchers treat a dataset as a cloud of points in a metric space and build a Vietoris–Rips filtration: they imagine a radius ϵ growing from zero, connecting points whose pairwise distance is at most ϵ, and watching which topological features appear and disappear. Each feature has a birth scale and a death scale, and the difference — its persistence lifetime — says how robust that feature is. Long lifetimes mean real structure; short ones mean noise.

From these lifetimes the authors build persistence weights by normalizing each lifetime by the sum of all lifetimes, then compute a Rényi persistence entropy of order q and its exponential, a PH-based Hill number, which they interpret as the effective number of topologically significant features. Setting q toward 1 recovers Shannon persistence entropy. The analysis is restricted to H₀ (connected components) and H₁ (loops). They argue the measure obeys four diversity principles — effective size, twin property, multi-scale, and symmetry — and prove each case in the appendix. For instance, duplicating a point adds a zero-length interval that dies immediately, so all nonzero persistence intervals and the final score stay the same.

For the experiments, they used a transfer-learning setup: fine-tuning BERT_base with an added dropout layer and softmax classifier, trained for 8 epochs at a learning rate of 1e-6 with a dropout rate of 10%. They evaluated on the Complaints dataset (TC), the SUBJectivity dataset (SUBJ), SentEval (SE), Arxiv-10, and Medical. From each dataset they built three balanced subsets — closest, farthest, and random — using each sample's maximum distance to all other points to rank it. Closest subsets came from the lower half of that ranking (core samples), farthest from the upper half (peripheral samples), and random subsets from all points regardless of ranking. Every training set contained 500 samples (250 per class), and each configuration was run three times per subset. They visualized subset structure in 2D with multidimensional scaling (MDS).

Why This Matters

The paper argues that data selection can be guided by measurable geometry rather than intuition, and that the widely used Vendi Score can point in the wrong direction when the goal is predicting training quality. If PH-based measures are trustworthy predictors, dataset construction becomes an optimization problem over topological structure rather than an exercise in collecting more data. The authors also note the analysis is limited to text classification with BERT fine-tuning, leaving other modalities and larger-scale settings to future work, so the claims are currently scoped to that setting.

Real-world applications implied by the work:

• Data augmentation: Deciding whether to shrink, expand, maintain, or shift the scope of a dataset when adding new samples.
• Data point selection and filtering: Prioritizing samples that yield well-separated clusters (high H₀ minimum) with stable loops (moderate H₁ mean) instead of redundant or peripheral ones.
• Synthetic data generation: Using the measures as a target signal for how much structural richness generated data should contain.
• Reducing dependence on large-scale data: Using smaller, better-selected subsets that approach full-dataset accuracy.

Industry relevance: Teams that fine-tune models on curated corpora could use these measures as a cheap pre-training check on dataset quality, since the computation depends only on the pairwise distance matrix and is invariant to sample ordering. This is especially relevant where annotation is expensive and duplicate-heavy data is common.

Future Directions

• Extending the evaluation beyond text classification with BERT fine-tuning to other modalities and larger-scale settings, which the authors explicitly leave open.
• Exploring how topological features can be directly leveraged to guide robust model training, improving generalization while reducing dependence on large-scale data.
• Building more deliberate augmentation strategies that explicitly target high H₀ minimum values combined with moderate H₁ mean lifetimes, rather than relying on random sampling to hit that balance.
• Investigating how the Hill-number order q can be tuned to emphasize rare versus dominant features in practical dataset construction, since the paper describes this as a tunable sensitivity rather than a solved design choice.

Target Audience

Researchers and practitioners working on data quality, dataset curation, and data augmentation for machine learning will benefit most, along with anyone applying topological data analysis to concrete ML problems. It is also useful for engineers who currently rely on entropy-based diversity scores and want to know where those scores break down. Readers without prior exposure to persistent homology will need the background section, but no deep topology training is required to follow the argument.

Authors’ abstract

High-quality training data is the foundation of machine learning and artificial intelligence, shaping how models learn and perform. Although much is known about what types of data are effective for training, the impact of the data's geometric structure on model performance remains largely underexplored. We propose that both the richness of representation and the elimination of redundancy within training data critically influence learning outcomes. To investigate this, we employ persistent homology to extract topological features from data within a metric space, thereby offering a principled way to quantify diversity beyond entropy-based measures. Our findings highlight persistent homology as a powerful tool for analyzing and enhancing the training data that drives AI systems.