Vectors, Inner Products, and Norms

A vector can represent an observation, a direction of motion, a set of model parameters, or a change to those parameters. Its meaning comes from the space and the operation it participates in. The same array may be a point when you compare examples and a direction when you apply a gradient update. Confusion begins when those roles are mixed without anyone saying so.

Vector geometry supplies three recurring ideas: direction, magnitude, and projection. Modern ML uses them everywhere, from cosine similarity in retrieval to norm penalties in optimization. The choice among them is not a matter of taste. As the sections below show, it has been proved about, measured, benchmarked, and written into a federal taxonomy.

The norm determines what counts as a large vector and which perturbations count as nearby. L1 adds absolute coordinate magnitudes, has a diamond-shaped unit ball in two dimensions, and often encourages sparse solutions. L2 uses Euclidean length, has a circular or spherical unit ball, is smooth away from the zero vector, and is rotation-invariant under orthogonal transformations. L∞ uses the largest absolute coordinate. Its unit ball is a square or a cube, and it controls the worst coordinate deviation while treating many smaller changes as irrelevant to the norm.

The claim that L1 is less dominated by one very large coordinate has an exponent attached to it. In 2001 Aggarwal and colleagues proved that the expected gap between the farthest and the nearest point under an L_k norm scales as d^(1/k - 1/2). Absolute contrast grows with dimension for L1, stays flat for L2, and vanishes for k >= 3. They concluded that L1 is preferable to L2 in high dimensions. That recommendation was then tested. Mirkes and colleagues ran it against 25 UCI databases analysed as 37 binary classification problems, comparing eight lp functionals, and reported in 2020: “A systematic comparison shows that the difference of the performance of kNN based on lp for p=2, 1, and 0.5 is statistically insignificant.” An asymptotic contrast argument and a measured k-NN win are different things.

L∞ is the least abstract column of the three, because an entire research field runs on a budget expressed in it. Madry and colleagues defined robustness against an explicit L-infinity ball in 2018, training and evaluating with epsilon = 0.3 on MNIST and epsilon = 8 on CIFAR-10. Their adversarially trained MNIST network scored 98.8% on natural examples. Under the strongest attack they ran, 100-step PGD with 20 restarts, it still scored 89.3%. The choice of norm is now codified by a government agency: “PGD can be applied to the ℓ2 and ℓ∞ distance metrics for measuring the perturbation of adversarial examples.” That is NIST AI 100-2e2025, the adversarial machine learning taxonomy published in March 2025, which cites Madry and colleagues as reference [232].

Different operations preserve different aspects of a representation. A norm measures magnitude according to a chosen geometry and answers how large. An inner product connects two directions, supports angle calculations, and answers how aligned. A distance compares points, depends on the chosen norm or metric, and answers how far apart. A projection extracts the component aligned with a subspace or vector and answers how much lies along this direction.

The four questions look interchangeable on a whiteboard. They are not. The d^(1/k - 1/2) scaling is a statement about distances. The epsilon = 0.3 ball is a statement about a norm. The ranking behaviour documented for OpenAI's embeddings below is a statement about an inner product under a unit-length constraint. Each result answers its own question, and none of them transfers for free.

Let u = (3, 4) and v = (4, 0). The projection of u onto v isolates the horizontal component.

Consider x = (3, 4), y = (6, 8), and z = (−4, 3). Three vectors are enough to separate what a norm sees from what an angle sees.

A long document sits far from a short one on the same topic, and the angle is what survives that difference. Microsoft's documentation of the Azure OpenAI embedding models gives the applied reason for preferring it: “This measurement is beneficial, because if two documents are far apart by Euclidean distance because of size, they could still have a smaller angle between them and therefore higher cosine similarity.” The x-and-y pair below is exactly that situation in two coordinates: a document twice as long, pointing the same way.

For vectors x and y, the dot product x·y sums coordinate-wise products. Geometrically, x·y = ||x||₂ ||y||₂ cos θ, so the result grows with both vector lengths and with directional agreement. A positive dot product indicates an acute angle, zero indicates orthogonality, and a negative value indicates opposing directions. In high-dimensional models these products often become logits, attention scores, or similarity signals. Because magnitude matters, a large-norm vector can dominate even when its direction is only moderately aligned. Cosine similarity removes that effect by dividing by both norms.

A vendor can also remove it in advance. OpenAI's embeddings developer guide says “We recommend cosine similarity”, and explains why the recommendation costs nothing: “OpenAI embeddings are normalized to length 1, which means that: Cosine similarity can be computed slightly faster using just a dot product”. With both norms equal to one, the factor ||x||₂ ||y||₂ disappears, and cosine similarity, the raw dot product and Euclidean distance produce identical rankings. The ambiguity in the worked example above is settled by the representation rather than by the metric. The same guide gives the default lengths, 1536 for text-embedding-3-small and 3072 for text-embedding-3-large, and notes that a text-embedding-3-large vector shortened to 256 dimensions still outperforms an unshortened 1536-dimension text-embedding-ada-002 on the MTEB benchmark.

The inequality behind that cosine has a tangled name. The integral form was published in 1859, in a monograph on inequalities between integrals, by Viktor Bunyakovsky. The Encyclopedia of Mathematics records the sequence plainly: “Bunyakovskii published his study as early as 1859, whereas in H.A. Schwarz’ work this inequality appeared as late as 1884 (without any reference to the work of Bunyakovskii)”. The MacTutor biography of Viktor Yakovlevich Bunyakovsky says the same: “He is best known for his discovery of the Cauchy - Schwarz inequality, published in a monograph in 1859 on inequalities between integrals. This is twenty-five years before Schwarz’s work.” Textbooks still hand the result to Cauchy and Schwarz alone.

Use the intended invariances to select the geometry rather than defaulting to Euclidean distance. First define the comparison: state whether magnitude, direction, worst-coordinate deviation, or total change matters. Then normalize deliberately, deciding whether vector length is signal, nuisance, or a consequence of frequency — unit length, OpenAI's answer for its own embeddings, is one defensible position and not the only one. Then inspect neighbors. Look at concrete nearest and farthest pairs, not only aggregate scores, because this is precisely where hubness hides: a Spearman correlation of 0.80 between intrinsic dimensionality and skewness is invisible in a mean recall number and obvious in a list of who keeps showing up. Then stress the metric by rescaling features, perturbing irrelevant coordinates, and comparing neighborhood stability. Finally, validate downstream: measure whether the chosen geometry improves the actual retrieval, clustering, or prediction task.

The last step is not a formality. A norm recommendation derived from a clean asymptotic argument about contrast survived until someone ran it over 25 UCI databases and 37 binary classification problems. At that point the difference between l0.5, l1 and l2 turned out to be statistically insignificant. Choose the geometry from the semantics; keep it only if the task agrees.