Structure Discovery Without Answer Keys

Ask how many groups are in a data set and the field hands you a shelf of answers. Thirty of them, in 1985. Milligan and Cooper ran a Monte Carlo evaluation of 30 procedures for determining the number of clusters, on artificial data sets built to contain either 2, 3, 4 or 5 distinct non-overlapping clusters. The true answer was known by construction. That is the easiest possible test. Their finding, in Psychometrika: “The simulation results for the stopping rules revealed a wide range in their ability to determine the correct number of clusters in the data.” They added a caution that has aged well, that “users are cautioned that the performance of some of the criteria may be data dependent”.

Thirty years later the question had not converged. NbClust, an R package released in 2014, opened by observing that “a wide variety of indices have been proposed to find the optimal number of clusters”, and it shipped accordingly: the package “provides 30 indices which determine the number of clusters in a data set”. Thirty procedures in 1985, thirty indices in 2014. An analyst still has to pick one and defend the pick.

That is the subject of this lesson. A clustering result depends on a feature definition, a scale, a distance, an algorithm and a stopping rule. No target label arrives afterwards to settle the argument. Unsupervised learning makes those choices visible precisely because nothing external adjudicates them.

Four layers stand between the raw records and whatever an analyst finally says the data means. Each of them is a decision, and any of them can alter the result.

Every named result in this lesson sits at one of these layers. The 30 stopping rules and the quality measures that answered Kleinberg are arguments about the structure rule. The distortion measured in two-dimensional embeddings of single-cell data is introduced at the representation layer. The sepsis phenotypes below are what happens at the fourth layer, when a grouping acquires a name.

Clustering, dimensionality reduction, and anomaly detection answer different questions even when they run on the same feature matrix. Two of the three have a published record of what their output cannot be asked to carry.

Compression to two dimensions is lossy, and the loss has been measured rather than merely suspected. Chari and Pachter said so in their 2023 abstract: “we show that extreme dimension reduction, from hundreds or thousands of dimensions to 2, inevitably induces significant distortion of high-dimensional datasets”. Their results put a number on it: the “correlation of cell type neighbor rankings to that of the ambient space were low (≤0.4) in PCA-preprocessed 2D embeddings, and at least 33% lower than those of the higher dimensional PCA spaces”. The rebuttal is the instructive part. A reply in the same journal the next year defended the pictures while conceding the distance claim outright: “while t-SNE and UMAP embeddings of single-cell data do not preserve high-dimensional distances, they can nevertheless provide biologically relevant information”. Both sides agree the picture does not preserve distances. They disagree about what the picture is still good for. That is this lesson’s own argument, running live in the literature.

An anomaly score, meanwhile, is worth only the review capacity behind it. The 2013 Target breach affected as many as 110 million customers. The Senate Commerce Committee’s majority staff report found that Target’s FireEye intrusion-detection system triggered urgent alerts on each installation of the data-exfiltration malware, and that “Target's security team neither reacted to the alarms nor allowed the FireEye software to automatically delete the malware in question”. On the generic “malware.binary” alert the staff wrote: “It is possible that Target staff could have viewed this alert as a false positive if the system was frequently alarming.” The detector fired. The ranking was correct. Nothing downstream of it happened. That is the failure mode a threshold argument is actually about.

The strongest projects connect a structural hypothesis to a concrete decision or investigation, and say in advance what would count as the hypothesis failing.

Use this sequence to turn a vague desire for “insights” into a testable unsupervised project. Step four is the one that gets skipped, and it is the one with a published cautionary tale behind it.

Stability is the reflex answer to how anyone knows the clusters are real, and stability can be manufactured by the procedure itself. Consensus clustering was run against data with nothing in it and produced groups anyway. Şenbabaoğlu and colleagues reported that in Scientific Reports in 2014: “Through simulations we find that CC is able to divide randomly generated unimodal data into apparently stable clusters for a range of K, essentially reporting chance partitions of cluster-less data.” The same paper found that “For data with known structure, the common implementations of CC perform poorly in identifying the true K”, and introduced the proportion of ambiguously clustered pairs, PAC, as a corrective measure.

The mechanism is now standard knowledge in the field’s tooling. M3C, a consensus-clustering package for Bioconductor, states it plainly in its vignette: the PAC score “has bias towards greater values of K. This is due to a general problem with this type of consensus clustering algorithm that occurs because as K increases the consensus matrix converges towards a matrix of perfect stability simply by chance.” That is why the fourth step reads predefine evidence rather than check stability. A stability number read off a run, with no null to compare it against, is exactly the number that randomly generated unimodal data also produces.

The goal is not to announce that the algorithm found hidden truth. The goal is a defensible representation, grouping, or ranking that helps a stated task — a narrower claim than “we discovered four segments”, and a far more useful one. Each result in this lesson is narrow in that way. The 30 procedures disagreed on synthetic data with a known answer, so the number of clusters is a choice you make and defend. Chari and Pachter measured how much a two-dimensional embedding loses, so a picture is evidence about neighbourhoods only up to a stated limit. The α to δ phenotypes were not reproduced one-to-one in other cohorts, so a label records a procedure and a population rather than a kind of patient. That stance makes later lessons easier. Every algorithm can be judged by the relation it preserves, the assumptions it imposes, and the evidence supporting its use.