Eigenvalues, Eigenvectors, and Spectral Thinking

Most vectors come out of a matrix with both a new length and a new direction. An eigenvector is special: the transformation leaves its line unchanged and only scales it, by an eigenvalue. These invariant directions are the matrix's internal modes. Repeated application amplifies the modes with large eigenvalue magnitude and suppresses the ones with small magnitude. Spectral thinking replaces a complicated coupled transformation with independent behavior along meaningful directions — when such a decomposition exists.

That last clause is not a formality, and this lesson keeps returning to it with figures. Eigenvalue analysis of plane Poiseuille flow predicts that the smooth flow first goes unstable at a critical Reynolds number of 5772·22. Orszag computed that value in 1971, and it is correct. Laboratory transition to turbulence is observed as low as about 1000. The spectrum was computed correctly and the prediction still did not match the experiment.

A simple iteration can estimate a dominant eigenvector, and every part of it has a cost somebody has measured.

Eigenvalues are not inherently about one task; their meaning comes from the matrix. Three matrices, three literatures, one operation — and in each, a measured consequence.

Covariance. Eigenfaces are, in Turk and Pentland's words, “the eigenvectors of the covariance matrix of the set of face images”. Their 1991 experiments used sixteen subjects and a database of over 2,500 images, digitized at all combinations of three head orientations, three head sizes and three lighting conditions. The scores split by what varied: “the system achieved approximately 96% correct classification averaged over lighting variation, 85% correct averaged over orientation variation, and 64% correct averaged over size variation.” Ranking directions by variance ranks them by whatever varies most, which is not always the thing you wanted.

The Fisherfaces paper measured what that costs. On the Yale database in 1997, close-cropped and leaving one out, the Eigenface method made 24.4% errors. Discarding the first three principal components — the lighting directions — brought that to 15.3%. Fisherfaces made 7.3%. The paper is careful about how far the trick goes: “Removing the largest three principal components does improve the performance of the Eigenface method in the presence of lighting variation, but does not achieve error rates as low as some of the other methods described here.” The three largest variances were the lamp, not the face.

Hessian. Curvature is not simply handed to you by the problem. Full-batch gradient descent manufactures its own, and in 2021 Cohen and four co-authors named the regime it actually operates in the Edge of Stability: “In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value 2/(step size), and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales.” A different group took that as its starting point in 2022, restating it as “Sharpness stabilizes around 2/LR and loss goes up and down across iterations, yet still with an overall downward trend”, and proved an implicit-regularisation mechanism for it, the associated flow minimizing λ1(∇²L). The step size fixes the curvature the optimizer will tolerate.

Graph Laplacian. The statement that zero eigenvalues relate to connected components is a theorem with a date. Fiedler argued in 1973, via Perron–Frobenius, that “it follows that a(G) is zero if and only if the graph G is not connected”, and he named a(G), the second smallest eigenvalue of the Laplacian, the algebraic connectivity of G. Von Luxburg's 2007 tutorial gives the general multiplicity version: “Then the multiplicity k of the eigenvalue 0 of L equals the number of connected components A1,...,Ak in the graph.” The eigenspace of 0 is spanned by the components' indicator vectors, under the ordering 0 = λ1 ≤ λ2 ≤ ... ≤ λn.

And “spectral clustering” has an address. Shi and Malik published normalized cuts in 2000, promising in the abstract that “We show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion.” The algorithm is two lines: “Solve (D − W)x = λDx for eigenvectors with the smallest eigenvalues” and “Use the eigenvector with second smallest eigenvalue to bipartition the graph”, at the splitting point that minimizes Ncut. Von Luxburg restates the same step as “Compute the first k generalized eigenvectors u1,...,uk of the generalized eigenproblem Lu = λDu”.

Read the eigenproblem, not the slogan. The paper does remark that the second smallest eigenvalue of D − W “is sometimes known as the Fiedler value”, but that is its account of earlier ratio-cut work. The Ncut algorithm uses the second smallest generalized eigenvector — equivalently an eigenvector of the random-walk normalized Laplacian — not the Fiedler vector of D − W.

If A has a suitable eigenbasis, x can be decomposed into eigenvector components and each component evolves independently. After k steps a component has been multiplied by λᵏ, so the largest magnitude wins whatever the starting vector was. The line |λ| = 1 separates growth from decay.

Epidemiology built a public threshold on exactly that line. The basic reproduction number is defined as this quantity and nothing else. Diekmann and two co-authors wrote it out in 1990: “The expected number of secondary cases produced by a typical infected individual during its entire period of infectiousness in a completely susceptible population is mathematically defined as the dominant eigenvalue of a positive linear operator.” Their threshold criterion is |λ| = 1 in words — “the disease can invade if R0 > 1, whereas it cannot if R0 < 1”. Van den Driessche and Watmough turned it into a matrix computation in 2002, setting “Ro = ρ(FV−1)” where “ρ(A) denotes the spectral radius of a matrix A”. They proved the disease-free equilibrium locally asymptotically stable when R0 < 1 and unstable when R0 > 1. R0 is not analogous to a dominant eigenvalue. It is one.

Do not interpret eigenvalues before identifying the matrix and the question. Name the operator. Check its structure for symmetry, stochasticity, sparsity or normality. Interpret what an invariant direction means in that domain, and examine gaps and signs. Then test sensitivity, because the last step is the one that has historically been skipped at the highest cost.

Hydrodynamic stability is the case study. Done accurately, eigenvalue analysis of plane Poiseuille flow gives a critical Reynolds number of 5772·22. Orszag published that figure in 1971, obtained with Chebyshev expansions and the QR matrix eigenvalue algorithm; his abstract states: “The method is applied to the stability of plane Poiseuille flow; it is found that the critical Reynolds number is 5772·22”. Transition to turbulence is observed as low as about 1000. Plane Couette flow is worse still: eigenvalue analysis predicts stability for all R, while transition is observed at R as low as about 350.

Trefethen and three co-authors opened their 1993 paper in Science by naming the failure: “This phenomenon has traditionally been investigated by linearizing the equations of flow and testing for unstable eigenvalues of the linearized problem, but the results of such investigations agree poorly in many cases with experiments.” Their resolution was pseudospectra. For these non-normal operators, small perturbations to the smooth flow can be amplified by factors on the order of 10^5 by a purely linear mechanism, even though every eigenmode decays monotonically. Nobody had computed the eigenvalues wrong. The eigenvalues were the wrong question. The check that catches it — perturb the operator, or look at singular values instead — is step five.

An eigenvalue with magnitude below one shrinks its mode under repeated application. Magnitude above one causes growth, while magnitude equal to one preserves amplitude. A negative real eigenvalue flips direction at each step. Complex eigenvalues represent rotation combined with scaling, and appear in oscillatory systems. For symmetric real matrices, eigenvalues are real and eigenvectors can be chosen orthonormal. The unnormalized graph Laplacian is the clean example, its spectrum ordered 0 = λ1 ≤ λ2 ≤ ... ≤ λn. That special case supports especially clean geometry.

Two working systems show how literally these boundaries are read. In epidemiology the |λ| = 1 line is the epidemic threshold: the dominant eigenvalue of the next-generation matrix is R0, and the disease “can invade if R0 > 1, whereas it cannot if R0 < 1”. In deep learning the boundary is not found in the problem but imposed by the optimizer. It drives the top Hessian eigenvalue up to the edge of its own stable region and parks it there at 2/(step size), a regime in which “loss goes up and down across iterations, yet still with an overall downward trend”.

Magnitude, sign and phase only carry these meanings when the eigenvalues describe the behaviour at all. For a non-normal operator the whole spectrum can sit safely inside the stable region, every eigenmode decaying monotonically, while the physical system becomes turbulent at a fifth of the predicted threshold. That vocabulary is complete for a normal matrix and partial for everything else.