Clustering and Segmentation: Useful Groups Without Invented Essences

Clustering methods favor different geometric stories. The reason no single story fits every dataset is a theorem, not a matter of taste.

Three elementary requirements each look reasonable on their own. Scale-Invariance is the requirement that f(d) = f(α·d) for any α > 0: changing the units of measurement must not change the partition. The other two are Richness and Consistency. In 2002 Jon Kleinberg proved that no method can hold all three at once: “For each n ≥ 2, there is no clustering function f that satisfies Scale-Invariance, Richness, and Consistency.”

So the four lenses below are not four tastes. Each is a way of giving up one of three properties nobody can keep together.

A retailer groups purchasing histories to plan campaigns. Several defensible segmentations answer different questions, and the data do not choose between them.

If one feature ranges from zero to one and another ranges into thousands, ordinary distance may be dominated by the larger numerical scale. Standardization changes the geometry and can change the clusters.

Which standardization is not a matter of habit. It has been measured, and the habitual answer lost. Milligan and Cooper compared eight standardization strategies in 1988, and the familiar z-score was not the winner: “approaches which standardize by division by the range of the variable gave consistently superior recovery of the underlying cluster structure”. An independent simulation study reproduced the result in 2020 — “Scaling with the range does fairly well, which is in line with the findings of Milligan and Cooper (1988)” — and reports that Steinley (2004) had reached the same conclusion for k-means.

Scaling is not a cosmetic preprocessing step. It expresses which differences should count as comparable, and domain-specific weighting may be more appropriate than automatic equalization.

The method should reflect plausible group shape, noise, scale, and the intended interpretation. The parameters below are set by people, not read off the data.

Take the density column's dependence on density parameters literally. DBSCAN arrived in 1996 with one of its two knobs set by fiat: “Therefore, we eliminate the parameter MinPts by setting it to 4 for all databases (for 2-dimensional data).” Eps itself was left to the user, to be read by eye off a sorted 4-dist graph. Twenty-one years later four of the same authors, joined by Erich Schubert, revisited the method and recommended minPts = 2·dim instead. On the UCI Household data they report that “there is no clearly visible 'valley,' 'knee,' or 'elbow' in this plot”, and settle on ε somewhere between roughly 500 and 2,000. scikit-learn, meanwhile, ships eps=0.5 and min_samples=5, its documentation saying of eps: “This is the most important DBSCAN parameter to choose appropriately for your data set and distance function.”

Three numbers, one algorithm, none of them supplied by the data.

Naming creates confidence. Evidence should come first — and step 5, the one most often skipped, is something you can actually measure.

Four sepsis phenotypes were derived by consensus k-means from 20,189 patients: α, n=6625 (33%); β, n=5512 (27%); γ, n=5385 (27%); δ, n=2667 (13%). They were validated in 43,086 more patients, with 28-day mortality of 5%, 13%, 24% and 40% respectively. Seymour and colleagues published that in JAMA in 2019. Then, instead of arguing about what the four groups are, they tested what the groups do: “In simulation models, the proportion of RCTs reporting benefit, harm, or no effect changed considerably (eg, varying the phenotype frequencies within an RCT of early goal-directed therapy changed the results from >33% chance of benefit to >60% chance of harm).”

Same treatment, same trial design, a different mix of clusters enrolled, and the verdict inverts. That is what validating the use looks like when it is done.

Organizations sometimes need groups for staffing, pricing, or communication. In that case, operational constraints can define a useful segmentation even when the data contain no natural clusters.

The clearest designed partitions carry a date. On 28 March 2024 the US Office of Management and Budget revised Statistical Policy Directive No. 15, which fixes the minimum race and ethnicity categories for federal reporting. It was the first revision since 1997. Five categories plus a separate ethnicity question became seven, collected in one combined question: “American Indian or Alaska Native, Asian, Black or African American, Hispanic or Latino, Middle Eastern or North African, Native Hawaiian or Pacific Islander, and White.” The revision adds “Middle Eastern or North African” as a category distinct from White. The US Department of Education describes the change to its own collections as “Shifting from the two-part race/ethnicity question to use a single combined race and ethnicity question that allows multiple responses”, to be complied with “no later than March 28, 2029”.

Nobody ran a clustering algorithm on the population and discovered a seventh group. A named body decided, on a stated day, where the lines go. Every federal collection has until 28 March 2029 to redraw them.

State whether the project is discovering structure or constructing a manageable partition. Confusing those aims can make arbitrary business bins appear scientifically inevitable.