Wireless Dataset Similarity: Measuring Distances in Supervised and Unsupervised Machine Learning

AI summary

Overview

Research area: Wireless communications and machine learning — specifically, measuring similarity (distance) between wireless datasets so that dataset choice, transfer, and augmentation can be reasoned about before training models.

Technical level: Advanced. The paper is built on formal distance definitions (geometric, statistical, subspace and manifold-based), latent-space projections via UMAP, and correlation analysis between distance matrices and transfer-performance matrices. It assumes familiarity with ML training/transfer concepts and with wireless channel data.

One-sentence scope: The paper proposes and evaluates a task- and model-aware framework for computing distances between wireless datasets, testing whether small distances predict good cross-dataset model transferability on an unsupervised CSI compression task and a supervised beam prediction task.

What This Paper Is About

Wireless ML research tends to focus on the learning method rather than the data, and real-world wireless data is hard to collect at scale, so practitioners rely heavily on simulated data (3GPP stochastic models, ray tracing tools such as Wireless InSite and SionnaRT, or hybrid approaches such as DeepMIMO) or on limited real datasets from testbeds like POWDER and AERPAW or datasets such as those from DeepSense6G and NYU. The problem is that there is no robust way to compare two datasets and predict whether a model trained on one will work on the other. The paper's goal is a dataset distance metric that correlates strongly with cross-dataset task performance, so that model generalization can be estimated without training.

Key Contributions

1. A task-driven, model-agnostic framework for evaluating similarity between datasets without training additional models.
2. Two distance metrics built on topological dimensionality reduction (UMAP): one applying Euclidean distance to KNN-formed clusters, and one applying Wasserstein distance between per-dimension distributions. Both are evaluated on supervised and unsupervised tasks.
3. A novel supervised, label-aware distance that uses label information and introduces penalty terms for dataset imbalance to refine datapoint comparisons.
4. A demonstration that dataset distances correlate with model performance, enabling task-specific metrics to guide dataset selection, decisions about model retraining, and benchmarking. The implementation, evaluation scripts, documentation, and reproducibility instructions are released open-source at the paper's project webpage.

Main Findings

• Unsupervised CSI compression: Using metrics based on UMAP embeddings combined with Wasserstein and Euclidean distances, the authors report Pearson correlations exceeding 0.85 between dataset distances and train-on-one/test-on-another task performance.
• Supervised beam prediction (downlink, CNN-based): A label-aware distance is derived by integrating supervised UMAP with penalties for dataset imbalance.
• Both tasks: The resulting distances outperform traditional baselines and consistently exhibit stronger correlations with model transferability.
• Practical implication: The framework is presented as a way to compare datasets before model training and to assess whether retraining is necessary for synthetic data or real-world data augmentation.
• Note on completeness: The provided paper content is truncated after Section VI (latent space projections with UMAP). Sections VII (unsupervised CSI compression results), IX (supervised LoS status and beam prediction results), and X (conclusions) are referenced in the organization section but their specific numbers, benchmark tables, and figures are not included in the content available here. Aside from the "exceeding 0.85" correlation, no numeric accuracy, loss, dataset-size, or model-performance figures are reported in the available text.

Methodology in Plain English

1. Define the problem formally. A dataset is a set of M vectors in R^N. A dataset distance is a function taking two datasets (which must share the same number of features N) and returning a non-negative number. The ideal distance is the one that maximizes the correlation between distances and measured model performance across many dataset pairs (their Equation 11).

2. Measure the ground truth. Train a model on each dataset, test it on every other dataset, and record the performance drop relative to testing on its own data. Stack these into a performance matrix; stack the pairwise distances into a distance matrix. Vectorize both and correlate them.

3. Choose how to compare datasets. Three families are discussed: matrix distances (limited, since they need equal numbers of datapoints and are sensitive to shuffling), pairwise geometric distances (need careful weighting in high dimensions), and distribution distances over per-feature histograms. Clustering before comparison adds robustness by averaging.

4. Project into a lower-dimensional space. High-dimensional data (e.g., a flattened 32×32 channel matrix is a 1024-dimensional vector) makes distribution estimation unreliable. The authors use UMAP, which builds a weighted k-nearest-neighbor graph and optimizes a low-dimensional embedding by minimizing cross-entropy between fuzzy topological representations in the high- and low-dimensional spaces. This is argued to preserve more global and local structure than PCA (linear), t-SNE (local-focused), or autoencoders (architecture-dependent).

5. Compute distances in the embedding. Euclidean distance over KNN-formed clusters, or Wasserstein distance between the per-dimension distributions.

6. Compare distributions fairly. For histogram-based distances, both datasets must share identical bin edges spanning their combined range, with the number of bins set heuristically as K = sqrt(M), where M is the larger of the two dataset sizes.

7. Evaluate on two tasks. Unsupervised CSI compression using an autoencoder architecture heavily inspired by CSINet+, and supervised downlink beam prediction using convolutional neural networks. Each is judged by how well its distance metric correlates with transferability.

Why This Matters

Impact on research: Wireless ML currently lacks "data operations" comparable to what other fields have — a way to understand, rank, and compare datasets. The paper argues this gap hinders progress toward large-scale generative models for wireless. By providing distances that correlate with transferability, it offers a pre-training way to reason about generalization, domain adaptation, and dataset selection, and it frames distance design itself as an optimizable, task-specific problem.

Real-world applications (as described in the paper):

• Dataset selection and augmentation: Supplementing limited real-world data with suitably similar simulated or existing datasets instead of generating new data from scratch.
• Simulation-to-real (sim2real) comparison: Judging how well a simulated dataset represents a target deployment.
• Synthetic data generation guidance: Producing task-specific synthetic data that is distributionally aligned with the target.
• Deciding on retraining or model switching: Predicting whether a model needs retraining before collecting new data, and detecting distribution shift during real-world operation.

Industry relevance: The authors include researchers from InterDigital alongside Arizona State University, reflecting direct interest from wireless industry players in reducing the cost of data collection and retraining. Because ray tracing a new environment is expensive and configuring stochastic models to match a specific environment is complex, a cheap distance metric that ranks existing datasets is commercially attractive for deployment and benchmarking.

Future Directions

• Completing the supervised and unsupervised evaluations: The organization section promises results on CSI compression, LoS status, and beam prediction, but the truncated content does not include the detailed tables behind the "exceeding 0.85" claim or the supervised beam prediction correlations.
• Extending to more tasks and metrics: The framing as an optimization problem (finding the distance function that maximizes correlation) invites searching over more distance families, embedding methods, and penalty designs — including label-aware variants beyond the imbalance penalty.
• Scaling and complexity: The paper raises the curse of dimensionality and the cost of histogram/PDF estimation; whether these distances remain reliable and tractable for very large or very high-dimensional wireless datasets is left open.
• Broader deployment questions: Using distances to detect distribution shift during real-world operation and to drive transfer learning and data augmentation policies remains a proposed application rather than something demonstrated in the available content.

Target Audience

Wireless communications researchers and engineers working on ML-based systems (channel estimation and compression, beam prediction, and related tasks), ML practitioners in telecommunications who need to choose or combine datasets, and researchers in domain adaptation and transfer learning interested in task-aware dataset distance metrics. Readers need comfort with dimensionality reduction, distributional distances, and neural network training to follow the formal sections, though the conceptual framing — "if two datasets are close, a model trained on one should work on the other" — is accessible to a broader technical audience.

Authors’ abstract

This paper introduces a task- and model-aware framework for measuring similarity between wireless datasets, enabling applications such as dataset selection/augmentation, simulation-to-real (sim2real) comparison, task-specific synthetic data generation, and informing decisions on model training/adaptation to new deployments. We evaluate candidate dataset distance metrics by how well they predict cross-dataset transferability: if two datasets have a small distance, a model trained on one should perform well on the other. We apply the framework on an unsupervised task, channel state information (CSI) compression, using autoencoders. Using metrics based on UMAP embeddings, combined with Wasserstein and Euclidean distances, we achieve Pearson correlations exceeding 0.85 between dataset distances and train-on-one/test-on-another task performance. We also apply the framework to a supervised beam prediction in the downlink using convolutional neural networks. For this task, we derive a label-aware distance by integrating supervised UMAP and penalties for dataset imbalance. Across both tasks, the resulting distances outperform traditional baselines and consistently exhibit stronger correlations with model transferability, supporting task-relevant comparisons between wireless datasets.