Distance, Similarity, and Neighborhoods

Which metric is useful depends on whether volume, composition, overlap, or correlated deviation matters. The gap between those questions has been measured. MTEB, the Massive Text Embedding Benchmark, put 33 models through 8 embedding tasks over 58 datasets and 112 languages. Its headline finding was not that one relation won. “We find that no particular text embedding method dominates across all tasks,” says its abstract, published in 2023. The measure that ranks first at semantic textual similarity is not the measure that ranks first at clustering or reranking. That is the practical content of the four columns below. They are not four styles of one measurement. They are four different questions, and 58 datasets say that being good at one of them does not make a system good at another.

Common measures answer different versions of the question “How alike are these observations?” Magnitude-sensitive geometry responds to absolute coordinate differences. Direction-sensitive geometry compares orientation and largely ignores length. Set-overlap measures look only at shared and unshared discrete items. Covariance-aware geometry rescales directions using an estimated covariance structure. Task-specific relations — edit distance, dynamic time warping, learned metrics, graph distances — encode specialized invariances. Reading down that list is the same as reading a list of assumptions you are about to make about your data.

A legal brief and a short email both mention “contract termination” five times. Their raw word counts differ sharply. Yet their normalized term directions may look similar. Distance is not a neutral ruler. It decides whether magnitude, direction, shared categories, correlated movement, or local connectivity counts as resemblance.

Testing deliberately chosen pairs is often more informative than inspecting an average score. One of these is not a thought experiment. It is a published measurement in which the data was held fixed and only the definition of similarity moved.

Design tests around pairs that should be close, pairs that should be far, and transformations that should not matter. Name the invariances first: state whether scale, translation, length, order, rotation, or shared absence should affect similarity. Build contrast pairs whose relationship domain experts can explain before seeing the metric. Normalize deliberately, according to feature semantics. Then inspect neighborhoods across dense, sparse, common and rare regions. Finally, stress the relation by perturbing irrelevant fields, and confirm that neighborhoods stay stable when they should.

The last two steps have a number attached to them, which makes them a test rather than an intention. Radovanović and colleagues named the failure mode in the Journal of Machine Learning Research in 2010. “Through theoretical and empirical analysis involving synthetic and real data sets we show that under commonly used assumptions this distribution becomes considerably skewed as dimensionality increases, causing the emergence of hubs, that is, points with very high k-occurrences which effectively represent "popular" nearest neighbors.” A hub is a record that turns up in everyone's neighbor list while treating almost none of them as its own neighbor. The relation has quietly stopped being symmetric.

That asymmetry is directly measurable. Take the k nearest neighbors of every point and count what fraction of those relations are returned in both directions. Roman Feldbauer and Arthur Flexer ran that count across an unusually wide sweep: “We evaluate the previously described methods on 50 different public machine learning data sets”. Their untreated starting point is “Baseline DS=0.50”. Half of all neighbor relations are one-way. Repairing the geometry moves that number, and by how much depends on the repair: “All local and global scaling methods consistently increase the number of symmetric neighborhood relations in nearly all cases (MP: DS=0.73, MPGaussI: DS=0.65, LS: DS=0.74, NICDM: DS=0.75)”, with localized centering and local DisSim both reaching 0.76. Run the same count on your own neighborhoods before you build anything on top of them. 0.50 is what an unaudited space looks like.

A metric determines which differences count, how dimensions trade off, and whether magnitude matters. That policy should be documented as clearly as a model architecture. Later clustering and anomaly methods inherit this geometry. If the neighborhoods are wrong, a sophisticated algorithm can only organize the wrong relationships more confidently.

The encouraging half of the evidence is that geometry is adjustable, and the adjustment can be measured. The Lk norm study did not stop at comparing Manhattan with Euclidean. It extended the norm to fractional values of k, and reported that those metrics gave “more meaningful results both from the theoretical and empirical perspective”. Fractional distances, its authors found, “can significantly improve the effectiveness of standard clustering algorithms such as the k-means algorithm”. The same is true of every choice in this lesson. 0.38 became 0.50 on an unchanged corpus of web pages. 0.50 became 0.76 on 50 data sets. And a threshold that varies in accuracy by factors of 10 to beyond 100 became, in one jurisdiction, a rule about when the system is not allowed to answer. A geometry is a hypothesis. It can be tested.