Stability, Consensus, and Reproducibility

A device cluster survives every random seed and bootstrap sample, and investigation reveals that all members came from one firmware version with a persistent logging offset. Stability confirms that the pattern repeats under tested perturbations. It does not determine whether the pattern reflects the intended phenomenon, a confounder, or a measurement defect.

That gap is not hypothetical. This lesson works through three published versions of it. A cluster-quality index driven to its maximum on data that contained no clusters. Five validated commercial tests disagreeing about 60.6% of the same 302 tumours. A 26-cell population that one of six replicates missed entirely, while it carried more than half the signal of the gene that defines the disease.

Different perturbations diagnose different failure modes.

Cluster IDs are arbitrary, so stability must be computed with label-invariant comparisons. The comparison itself needs a baseline, and the unadjusted measures do not have one. Vinh and colleagues showed that in 2010. Averaging over 200 random clusterings and all 19,900 pairs of them, mean MI, NMI and RI drift upward as K grows, VI drifts downward, and only ARI stays close to zero. Take a 100-item data set compared by two 10-cluster partitions. The expected mutual information is E{I(U,V)} = 0.4618, against a maximum of log(10) = 2.3. Roughly a fifth of the available scale is handed out for nothing, so “correction for chance is needed since the baseline is high”. They state the consequence plainly: “This is clearly an unwanted effect, since a consensus index built upon the MI, NMI and VI would be biased in favour of a larger number of clusters.” The bias is a function of how thinly the data are cut. At 1,000 items and 10 clusters, N/K = 100, the same bounds fall to 0.0764 and 0.0780, “small enough for many applications, therefore adjustment for chance might be omitted”. That is the operational rule. Around a hundred points per cluster, chance stops flattering you.

A chance-adjusted global score still tells you nothing about which group moved, and that is where instability actually lives. OPTIMA prelim randomised 313 women with early breast cancer; 302 of them had samples available for multiparameter testing. Five commercial multiparameter tests were run on those same 302 tumours. Bartlett and colleagues report the result in the abstract: “Only 119 (39.4%) tumors were classified uniformly as either low/intermediate risk or high risk, and 183 (60.6%) were assigned to different risk categories by different tests, although 94 (31.1%) showed agreement between four of five tests.” Pairwise kappas ran from 0.33 (MammaPrint vs IHC4) to 0.60 (IHC4 vs IHC4-AQUA). The three subtype callers looked closer still. Each assigned 59.5–62.4% of tumours to luminal A, an aggregate profile so similar that a summary statistic reads as near-agreement. They gave different subtypes to 123 tumours (40.7%). The marginal distributions matched; the patients did not.

Adjusted Rand index compares whether pairs are grouped together, adjusted for chance. It is invariant to cluster label names, defined for hard partitions, usable across different cluster counts, and silent about which groups changed. Adjusted mutual information measures shared information between partitions with the same chance correction. It handles label permutations and summarizes global agreement, but it can obscure local cluster instability and depends on partition granularity. Cluster matching aligns individual clusters using overlap, centroids, or assignment optimization. It locates the stable and unstable groups, needs a matching rule, can leave splits and merges ambiguous, and supports profile-level comparison. Only the third kind of comparison could have found OPTIMA's 123 reassigned tumours.

A robust result should be tested against more than random initialization. Optimization variation comes from seeds and initialization changing which local solution is reached. Sampling variation comes from bootstrap or subsampling changing which observations support the structure. Representation variation comes from features, scaling, missingness, and embeddings altering the geometry. Method variation comes from different algorithms rewarding different shapes, densities, and partitions. Temporal variation comes from later data introducing drift, new regimes, or changed measurement.

The last two have been measured rather than assumed. Duò and colleagues evaluated 14 R clustering algorithms across nine public scRNA-seq data sets and three simulations, and found substantial differences in performance, run time and stability between the methods. Krzak and colleagues evaluated 13 methods across 17 real and 25 simulated data sets, and attributed the great variability in performance to user-chosen parameter settings; most method and parameter combinations systematically under- or overestimated the number of clusters. A perturbation suite that varies only the seed is testing the one source of variation these benchmarks found least of.

Global agreement should be paired with group-level and example-level diagnostics. First, define the perturbation suite: seeds, resamples, feature blocks, preprocessing variants, methods, and time windows. That is the Ben-Hur principle — perturb the data, compare the partitions that come back. Second, fit an ensemble, storing assignments, parameters, profiles, and diagnostics for every run. Third, compare globally with label-invariant, chance-adjusted agreement and cluster-count summaries; the unadjusted baseline is only negligible once N/K is around 100. Fourth, align groups to identify persistent clusters, splits, merges, and unmatched rare groups. The 40.7% of OPTIMA tumours that moved between subtype callers are invisible to any step that stops at step three. Fifth, inspect unstable points: observations that switch repeatedly or remain unassigned. Sixth, decide the response — keep alternatives, simplify the claim, collect data, or build a consensus partition, in the knowledge that consensus typically did not beat the best combined method in the benchmark that tested it.

A trustworthy analysis does not need every observation to retain one label across every run. It needs to show which structures persist and where the evidence remains weak. A 26-cell cluster that one replicate out of six failed to contain is not a failure of the report; suppressing it would be.

Store stability artifacts with the cluster version. Downstream users should know whether a label belongs to a robust core, a contested boundary, or a transient regime. Where the groups feed a decision about individuals, they should also know whether the agreement being quoted is agreement about proportions or agreement about people.