Evaluating Unsupervised Structure Without Labels

An average silhouette of 0.88 sounds like a clean partition. It can also be eight objects, one of them a distant outlier, and nothing else going on. Peter J. Rousseeuw printed that example in 1987, in the paper that introduced the silhouette. The outlier sits alone, where his own convention sets s(i)=0 for a singleton cluster. Every remaining object is so far from that lone point that its silhouette approaches the maximum. The overall average silhouette width lands at 0.88 — in general (n-1)/n. Nothing about the other objects has to form a real group for that number to appear.

“Therefore, one should never merely accept a high overall average silhouette width at its face value, but also look at the graphical output itself to find out what caused it.” — Peter J. Rousseeuw, 1987

The score measured geometry under the chosen distance. It did not measure whether the partition supported a decision, represented stable behavior, or aligned with available interventions. A team that selects the clustering with the highest silhouette has selected for that geometry and for nothing else — including for whichever artifact happened to produce it.

Disagreement is diagnostic. Each measure sees a different property, and the published record contains cases where the measures split on the same data and the split was the finding.

Each layer answers a different question. Each can disagree with the others. Stability is the layer most often described in the vaguest terms, so it is worth seeing what a published pass mark actually looks like. Prediction strength, from Tibshirani and Walther in 2005, validates a clustering by treating it as a supervised classification problem on held-out data. The reference manual for the R package fpc states the rule in one sentence: “The prediction strength is defined according to Tibshirani and Walther (2005), who recommend to choose as optimal number of cluster the largest number of clusters that leads to a prediction strength above 0.8 or 0.9.”

Koren and colleagues adopted it over the silhouette index as their primary absolute measure in the enterotype meta-analysis: “we recommend using at least one absolute measure (specifically, we recommend PS)”. That is what every layer of the portfolio should look like when it is written down. A named method, a number, a published threshold, and a stated blind spot. A layer that cannot be reported in that form is an impression, not evidence.

An evaluation report should expose tradeoffs rather than hide them behind one ranking. Step 3 is the step that usually stays abstract, so make it a computation with a cut-off. Christian Hennig's cluster-wise bootstrap, published in 2007, scores each individual cluster by the mean Jaccard similarity between it and the most similar cluster found in bootstrap resamples. A value at or below 0.5 marks a dissolved cluster. The thresholds above it are published with the implementation, the clusterboot() function of the R package fpc: “Generally, a valid, stable cluster should yield a mean Jaccard similarity value of 0.75 or more”, and “'Highly stable' clusters should yield average Jaccard similarities of 0.85 and above.”

Most real results land in the band between, and the manual is explicit about what that band means: “Between 0.6 and 0.75, clusters may be considered as indicating patterns in the data, but which points exactly should belong to these clusters is highly doubtful.” Report one number per cluster instead of one number per solution. That is the point of the procedure. It is what lets step 6 bound the claim to the two clusters that survived rather than to the partition as a whole.

Internal indices are useful diagnostics. But their preferences can favor the wrong structure for a domain, and their definitions carry conditions that are easy to lose in a leaderboard. The silhouette is the clearest case, because its author published the formula and the caveat together. Rousseeuw defined s(i) from the within-cluster dissimilarity a(i) and the nearest-other-cluster dissimilarity b(i) as (b(i)-a(i))/max{a(i),b(i)}. The construction requires more than one cluster, and he set s(i)=0 for a singleton cluster by convention.

Anyone calling the metric today gets the same limits from the scikit-learn reference for silhouette_score: “The Silhouette Coefficient for a sample is (b - a) / max(a, b)”, “The best value is 1 and the worst value is -1”, “Values near 0 indicate overlapping clusters”, and “Note that Silhouette Coefficient is only defined if number of labels is 2 <= n_labels <= n_samples - 1”. Every one of those statements describes the geometry of a distance matrix. None of them describes whether the groups mean anything. That is why the same number can be produced by well-separated populations and by a single distant point, and why the silhouette, Davies–Bouldin, Calinski–Harabasz and external-agreement panels below have to be read as four different questions rather than four attempts at one.

The absence of labels does not make evaluation impossible. It changes the burden of proof from matching one answer key to assembling convergent evidence. A mature report can conclude that a structure is stable but not useful, useful but unstable, or promising only within a defined population. Those narrower conclusions support better decisions.

Agreement scores need a baseline before they can be read, and the Rand index has none. Vinh and colleagues show that its baseline “can be high and does not take on a constant value”, drifting with the number of clusters. Hubert and Arabie's correction fixes it: “The ARI is bounded above by 1, and equals 0 when the RI equals its expected value”.

That is the shape of an honest unsupervised report. Each measure named, each baseline stated, and the disagreements left visible rather than resolved by choosing a favourite. When Koren and colleagues write that “Together these results indicate that these different clustering methodologies can yield inconsistent results”, they are reporting a property of the data and the measures, not a defect in the write-up.