Dimensionality Reduction and Representation Discovery

A dimensionality-reduction method maps many measured variables into fewer coordinates. The new coordinates may compress redundancy, expose factors, speed downstream models, or support visualization.

Every reduction sacrifices information. The question is not whether the map is smaller. It is which relationships the map preserves and which it distorts — and, for at least one of those relationships, exactly what the preservation costs.

Preservation goals conflict. That is why no low-dimensional representation is universal. Reconstruction retains enough information to approximately recover the original observation. Variance preserves directions with large spread under a linear projection. Neighborhood keeps selected local relationships close in the new space. Task signal retains what a later prediction, retrieval, or control objective needs. A method optimizing one of these is not thereby serving the others.

One of the four goals has an exact, proved price. Take n points in any Euclidean space. They can be mapped into k dimensions with every pairwise distance preserved to within a factor of (1 ± ε), provided k ≥ 4(ε²/2 − ε³/3)⁻¹ ln n. That is the Johnson–Lindenstrauss lemma, in the form Dasgupta and Gupta proved. The remarkable part of the bound is what is missing from it. Here k depends on log n and on ε only. It does not depend on the original number of features at all. Ten thousand columns or ten million: the dimensions you need to hold all pairwise distances to within 1 ± ε are set by how many points you have and how much distortion you will accept.

And the price cannot be haggled down. In 2016 Larsen and Nelson proved a matching lower bound of m = Ω(ε⁻² lg n). “This lower bound matches the upper bound given by the Johnson-Lindenstrauss lemma,” they write. No cleverer map, linear or not, does better. This is what a preservation goal looks like when it is stated precisely enough to be settled: a named quantity, an exchange rate in dimensions, and a proof that the rate is the best available. The other three goals in this lesson have no such receipt. Arguments about them are settled by measurement, not by theorem.

One mathematical operation can serve very different practical purposes. Two results show the range.

Two coordinates out of half a million can locate a European on a map. Novembre and colleagues ran principal component analysis on 3,000 European individuals genotyped at over half a million DNA sites, and published the result in Nature in 2008. The first two linear coordinates reproduce the map of Europe. They reproduce it closely enough to place a person on it. The paper says: “Using a multiple-regression-based assignment approach, one can place 50% of individuals within 310 km of their reported origin and 90% within 700 km of their origin.” Two years later an independent group reproduced the effect with panels of only 500 or 1,000 selected SNPs, still predicting individual origin to within a few hundred kilometres. When a reduction is keeping something real, this is what it looks like: an external, checkable quantity recovered from the new coordinates.

Compression can also be a dial rather than a fixed choice. In 2022 Kusupati and colleagues published Matryoshka Representation Learning: one embedding whose leading prefixes are themselves usable representations. They report up to 14× smaller embedding size at the same ImageNet-1K accuracy, and up to 14× real-world speed-ups for large-scale retrieval. The technique is in production. Google's Gemini embedding models are trained with it, and the Gemini API documentation tells the caller what that buys: “By default, both models output a 3072-dimensional embedding, but you can truncate it to a smaller size without losing quality to save storage space.” You pick 3,072, 1,536 or 768 dimensions at call time. A reduction decision has moved out of the research paper and into an API parameter a reader can go and use today.

The method family changes which relationships are easy to preserve and interpret.

A linear projection represents every coordinate as a weighted combination of original features. It is often easier to inspect, and it supports reconstruction and variance analysis. It cannot unfold every curved structure, and it is sensitive to scaling. It is what recovered the map of Europe from half a million genotyped sites.

A nonlinear embedding allows a curved or local map into fewer dimensions. It can reveal local manifolds. But global geometry may distort, parameters affect the picture, and the axes are rarely directly interpretable.

A task-trained representation learns a space for a downstream objective. It preserves target-relevant signal, may discard other useful information, can encode target bias, and requires protected downstream tests.

Evaluation depends on why the representation was created. Reconstruction evidence measures how much original information can be recovered. It is useful for compression, but it may preserve irrelevant detail, is sensitive to feature scaling, and guarantees no task utility. Neighborhood evidence checks whether meaningful nearby relationships survive. It suits retrieval and local exploration, though the choice of neighborhood matters, global distortion can hide inside it, and it needs a domain-relevant notion of similarity. Downstream evidence tests the representation in a separate task or decision. It connects the features to practical use, requires protected evaluation, can expose lost signal, and may overfit if repeatedly tuned.

The third column now has a public, sized instrument. The Massive Text Embedding Benchmark, published in 2022, spans 8 embedding tasks over 58 datasets and 112 languages, and it benchmarked 33 models across all of them. Its abstract states the result that this lesson's preservation-goals section only asserts: “We find that no particular text embedding method dominates across all tasks.” A representation good for semantic similarity is not thereby good for clustering, retrieval or reranking. It took 33 models over 58 datasets to say so with evidence rather than intuition.

Production APIs have absorbed the finding. Google's Gemini embeddings API exposes a task_type parameter. The interface asks the caller which task the vector is for, because the vector should differ depending on the answer.

Treat the space as a model. A lower-dimensional space carries assumptions and needs evaluation.

First, name the purpose: compression, visualization, retrieval, denoising, or prediction. Second, prepare the inputs. Review scaling, missingness, sparsity, and dominant nuisance variables. Third, define preservation explicitly — reconstruction, variance, neighborhoods, or task signal — because only the first of these comes with a proved exchange rate in dimensions. Fourth, test perturbations. Vary samples, seeds, dimensions, and reasonable method settings, since random Gaussian clouds can clump convincingly at low perplexity. Fifth, inspect correspondences. Trace points and neighborhoods back to original examples. Sixth, validate downstream: measure whether the representation helps the intended use without contaminating evaluation.

Nearby vectors indicate similarity according to the data and objective that produced the space. That is a narrower claim than it sounds. Proximity does not guarantee shared cause, moral equivalence, or human-understood semantics.

Embedding geometry can also encode frequency, language, culture, and collection bias, and a national measurement agency has quantified what that does to a fixed distance threshold. In December 2019 NIST ran 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. The finding concerns false positives: the rate at which two different people's face vectors fall inside the same match threshold. The executive summary of that report states it in one line: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” In the higher-quality application photos the rate was highest for West and East African and East Asian faces, and lowest for Eastern European faces. False negatives varied by factors below 3. The U.S. Commission on Civil Rights restated the same finding in its September 2024 report.

One threshold, one embedding space, and an error rate that differs by two orders of magnitude depending on whose face is in it. Document the training context before treating vector distance as a universal meaning measure.