Projection Traps, Trustworthiness, and Joint Workflows

Every preservation score has a target, and picking one without knowing its target hides the distortion that matters most. In 2003 Kaski and five colleagues gave gene expression profiles to “hierarchical clustering, multidimensional scaling, and the self-organizing map”, then asked whether the resulting pictures could be believed: “if two samples are visualized to be similar, are they really similar?” That question is what trustworthiness scores. Their warning is about attention as much as error: “The proximities that are visible on the display are salient, and if they are not trustworthy the whole display is misleading.” The wrong proximity is the one the eye goes to first.

How much the choice of score decides is visible in a published disagreement about the same embeddings. Chari and Pachter built Picasso, an autoencoder that forces any single-cell dataset into an arbitrary two-dimensional shape — an elephant, for instance. On their distance-based metrics the elephant came out no worse than t-SNE and UMAP, and they argued the popular embeddings were no less arbitrary than it. In 2024 Lause and colleagues re-ran the comparison with neighbourhood- and class-preservation metrics. The ordering reversed: “In both metrics, UMAP and t-SNE consistently and strongly outperformed PCA and Picasso elephant embeddings (Fig 2C–2D, >90% vs. <62% accuracy, >15% vs. <5% recall for all datasets).”

Two groups, the same objects, opposite verdicts, and no arithmetic error on either side. A distance-based score rates an elephant the equal of UMAP. Neighbourhood and class scores separate them by >90% against <62% accuracy and >15% against <5% recall, on every dataset tested. The later group also confirms the earlier one on exactly the property the earlier metrics were built to measure: on real single-cell data, t-SNE and UMAP embeddings “do not preserve high-dimensional distances”. Both statements are true at once because they answer different questions. Choosing a metric is choosing which failure you will be able to see. That choice belongs in the report, next to the number.

Two published experiments begin with data that contains no groups at all, and end with groups.

The first fed t-SNE nothing but noise. The 2016 Distill article on using t-SNE describes the input exactly: “The next diagrams show genuinely random data, 500 points drawn from a unit Gaussian distribution in 100 dimensions.” Under the heading “Random noise doesn't always look random”, Wattenberg and colleagues report what the low-perplexity run produced: “The plot with perplexity 2 seems to show dramatic clusters.” Nothing in the source distribution corresponds to them.

The clustering half of the pipeline fails the same way on its own. A paper on selective inference for hierarchical clustering, published in the Journal of the American Statistical Association in 2024, opens with 100 observations drawn from a single two-dimensional Gaussian with mean 0 and variance 1. Average-linkage hierarchical clustering splits them into three clusters, and the empirical means of those three clusters differ substantially. One population went in. Three groups with visibly different centres came out.

An analyst who reduces 768-dimensional document embeddings to two coordinates, clusters the map and reports five themes has done nothing that those two experiments did not do to noise. The themes may be real. The pipeline that produced them is not evidence that they are. The identical pipeline produces the same kind of output when there is nothing there, so the evidence has to be tested at each transformation boundary.

These artifacts come from objectives, sampling, display and human perception, not from obvious software bugs. Several of them were demonstrated on controlled synthetic data in that 2016 Distill article. Its section on distances between clusters ends with a rule rather than a caveat: “The basic message is that distances between well-separated clusters in a t-SNE plot may mean nothing.” That is not a claim about badly tuned runs. Measurements on real single-cell data in 2024 arrive at the same place from the other direction: t-SNE and UMAP embeddings “do not preserve high-dimensional distances”, even where those embeddings are strong on neighbourhood and class preservation.

No single metric captures all distortions. That is why one distance-based score and one neighbourhood score can order the same four embeddings differently and both be correct. Evaluation must match the intended use, and the intended use has to be written down before the score is chosen.

A staged workflow reduces the chance that an attractive map becomes its own proof. It does not by itself buy back valid inference. Hold out a second independent sample, test the clusters found in the first, and the Type I error rate is still extremely inflated. The confirmatory stage therefore has to bring evidence of a different kind: source-space stability, a second reducer, raw examples read by someone who knows the domain, and the downstream decision. Fresh rows for the same test are not enough.

Dimensionality reduction is valuable because it discards information deliberately. The professional question is whether it discards the information the decision can afford to lose, and that question has an answer only once the relation of interest is named. Snyder's cartographer chooses the characteristic to show accurately, and says which one it is. The equivalent sentence is missing from most embedding plots.

So report preservation metrics beside the visualization. State which score was used, and therefore which failure it can see. Inspect the affected slices. Keep the original observations available. The goal is not a flawless map but a bounded and testable representation — one whose claims do not rest, as 42% of one year's fMRI papers did, on evidence taken from inside the selection that produced them.