Unsupervised learning
PCA: Variance, Components, and SVD Intuition
Understand principal components, scores, loadings, variance, centering, and the connection between PCA and singular value decomposition.
By the end you can
- Explain PCA as a linear projection onto orthogonal directions of decreasing variance
- Distinguish observations scores from feature loadings and singular values
- Describe why centering is essential and scaling is a separate modeling choice
- Connect PCA projection with SVD without treating them as identical concepts
Example
Reading components without inventing a story
Loadings and scores support interpretation when scale, sign, and uncertainty stay visible. Each case below has a published record behind it. Two of them are cases where a leading component was read as biology and turned out to be something else.
- Correlated sensors: The first component loads positively on temperature and pressure, summarizing their shared movement rather than naming a cause for it.
- Contrast direction: A later component that loads positively on one feature and negatively on another looks like a tradeoff. It may instead be a redundant curve of the component before it. The shape is called a horseshoe, and it was analysed in 2008 in the Annals of Applied Statistics on the 2005 US House of Representatives roll-call votes: “Certain MDS and kernel projections output “horseshoes” that are characteristic of dimensionality reduction techniques. We show that, in general, a latent ordering of the data gives rise to these patterns when one only has local information.” Diaconis and colleagues also warn that the second eigenvector is not a quadratic — or any — function of the first. The two coordinates trace a parametrized curve that cannot be written in functional form. A later paper on genetic data reports the same shape in principal-component plots: “PC plots are indeed expected to exhibit horseshoe effects, an artifact in which the second axis is curved relative to the first axis”. A curve is not always a contrast.
- Sign ambiguity: Multiplying one loading vector and its scores by minus one leaves the PCA solution equivalent. The field treats that freedom as a hazard, not a curiosity. A 2007 Sandia National Laboratories technical report opens with it: “Though the SVD and EVD are well-established and can be computed via state-of-the-art algorithms, it is not commonly mentioned that there is an intrinsic sign indeterminacy that can significantly impact the conclusions and interpretations drawn from their results.” scikit-learn settles it in code. The library ships svd_flip, a “Sign correction to ensure deterministic output from SVD” that “Adjusts the columns of u and the rows of v such that the loadings in the columns in u that are largest in absolute value are always positive”. The sign you see was chosen by a convention, not by the data.
- Low-variance event: A rare failure signature can appear in a later component despite high operational importance, because the ordering ranks variance and nothing else. The reverse also holds: in the POPRES analysis described in the next section, the two components that reconstruct the geography of Europe carry only about 0.30% and 0.15% of the total variance.
- Batch effect: The first components can separate laboratories or devices rather than biology or behavior. Lin and colleagues reported in PNAS in 2014 that “we found that the mouse and human samples cluster by species when the data are projected onto the first three principal components”. Gilad and Mizrahi-Man re-ran that PCA in R with prcomp (center=TRUE, scale=TRUE), on 14,744 orthologous gene pairs across 26 samples, and published the reanalysis in F1000Research in 2015: “Here we show that the Mouse ENCODE gene expression data were collected using a flawed study design, which confounded sequencing batch (namely, the assignment of samples to sequencing flowcells and lanes) with species. When we account for the batch effect, the corrected comparative gene expression data from human and mouse tend to cluster by tissue, not by species.” After correction the first five PCs — 56% of the variability together — no longer supported species clustering. The species signal was a sequencing batch.
A cloud that became simple after a rotation
In 2008 a change of coordinates redrew a continent. Novembre and colleagues ran PCA on 197,146 SNP loci in 1,387 European individuals from the POPRES sample, then plotted the first two components against each other. The scatter was a map of Europe. Their abstract in Nature states the result: “Despite low average levels of genetic differentiation among Europeans, we find a close correspondence between genetic and geographic distances; indeed, a geographical map of Europe arises naturally as an efficient two-dimensional summary of genetic variation in Europeans.”
The two directions that did this carry almost none of the variation. PC1 accounted for about 0.30% of the variance and PC2 for about 0.15%. The first eigenvalue was 4.09, the second 2.04. A regression on those two components placed 50% of individuals within 310 km of their reported origin, and 90% within 700 km. A different group reached the same conclusion on a different sample and platform — 3,112 individuals, more than 270,000 Illumina SNPs — reporting that “the genetic structure of the European population correlates closely with geography”.
PCA changes coordinates. It did not find migration, ancestry, or a causal driver. It summarized linear variation in the centered dataset, and the geography was already in there. Which is also why a variance share of 0.30% tells you nothing about whether a direction matters.
Principal components are directions of variance, not hidden causes.
Analogy
Rotating a camera to align with the longest shadow
Photographing an elongated object wastes most of the frame until the camera is turned. Rotate it until one axis follows the object's longest direction and the coordinate description becomes compact.
Physical volume is not PCA's objective. The method follows statistical variation after centering and any chosen scaling. A dominant batch effect can therefore occupy the “largest direction” without representing the phenomenon of interest. That is what the Mouse ENCODE reanalysis found: the leading directions tracked sequencing flowcells and lanes.
PCA aligns coordinates with variation, not necessarily with meaning.
Key idea
Centering and scaling answer different questions
PCA requires centering for its standard covariance-based interpretation. Scaling each feature to unit variance is optional, and it does not rescale the components — it replaces them. Jolliffe and Cadima say so without hedging in their 2016 Royal Society review: “Such correlation matrix PCs are not the same as, nor are they directly related to, the covariance matrix PCs defined previously.” The explained-variance profile differs as well, often needing more components to reach the same percentage.
The size of that choice is measurable on a public dataset. scikit-learn's own “Importance of Feature Scaling” example runs the UCI Wine recognition data — 178 instances, 13 chemical features — through two pipelines that differ in one step. A LogisticRegressionCV trained on two unscaled PCA components reaches 35.19% test accuracy at a log-loss of 1.18. Insert StandardScaler before the PCA and the same pipeline reaches 96.30% at a log-loss of 0.0739. One column explains the gap: “Indeed we find that the “proline” feature dominates the direction of the first principal component without scaling, being about two orders of magnitude above the other features.” Same data, same model, one extra step.
Standard tools encode the split rather than leaving it to the analyst's memory, and the wording is explicit. scikit-learn documents that “the input data is centered but not scaled for each feature before applying the SVD”, and points anyone who wants an uncentered decomposition to the separate TruncatedSVD class, “where the data are not centered”. R makes scaling an argument with a stated default and a stated opinion: “The default is FALSE for consistency with S, but in general scaling is advisable.” Choose it from units and semantics. A feature measured in dollars should not dominate merely because its numerical range is larger. Equal variance is not the right policy every time either.
Standardization is a modeling decision layered on top of PCA.
Comparison
PCA as projection and SVD as matrix factorization
The connection is close. The analytical interpretations are not interchangeable. Jolliffe and Cadima derive both routes and treat them as one result: PCA “can be obtained as the solution to an eigenproblem or, alternatively, from the singular value decomposition (SVD) of the (centred) data matrix”. Their equation (2.2) is the object that “links up the eigendecomposition of the covariance matrix S with the singular value decomposition of the column-centred data matrix”.
R states which route it takes in its own manual. The prcomp function works “by a singular value decomposition of the (centered and possibly scaled) data matrix, not by using eigen on the covariance matrix”, and that route is “generally the preferred method for numerical accuracy”. The two descriptions agree on the answer and differ on how it is reached. They also differ on what else they cover. An SVD by itself imposes no statistical interpretation and carries no centering convention. That is exactly why scikit-learn keeps the uncentered case in a different class.
PCA view
Seek orthogonal directions maximizing variance of centered observations.
- Provides explained-variance interpretation
- Defines components in feature space
- Requires a centering convention
- Supports dimensionality reduction
SVD view
Factor a matrix into left singular vectors, singular values, and right singular vectors.
- Applies to general matrices
- Provides numerical algorithms for PCA
- Separates observation and feature directions
- Does not by itself impose a statistical interpretation
Truncated approximation
Keep only leading singular directions.
- Produces a low-rank matrix
- Minimizes squared reconstruction error under standard conditions
- Can discard rare low-variance signals
- Depends on the retained rank
Visual
The objects produced by PCA
Several related quantities play different roles in interpretation and transformation. Reported percentages of explained variance come from the singular values. That is why a component can hold a very small share — 0.30% in the POPRES analysis — and still carry the structure an analyst cares about.
Centered data matrix
Original observations after subtracting each feature mean.
Loading vectors
Orthogonal feature-space directions defining the components.
Scores
Coordinates of observations after projection onto the loading vectors.
Singular values
Quantities related to variance captured along the components.
Reconstruction
Approximation obtained by mapping retained scores back into the original feature space.
Loadings describe directions; scores locate observations along those directions.
Steps
Build an interpretable first PCA analysis
Inspect preprocessing, variance, loadings, scores, and reconstruction together. Steps one and four are what the Mouse ENCODE reanalysis was. Someone asked which flowcell and lane each sample had been sequenced on, then looked at what the leading components were actually separating. That audit reversed the published conclusion, from clustering by species to clustering by tissue.
1. Audit features
Review units, missingness, outliers, transformations, and dependence between observations.
2. Choose centering and scaling
Document whether PCA uses covariance or correlation-style geometry.
3. Fit several component counts
Record explained variance and reconstruction behavior without using one cutoff blindly.
4. Inspect loadings and scores
Review dominant features, extreme observations, batches, and sign conventions.
5. Test stability
Repeat the factorization under resampling and preprocessing changes, then align comparable component subspaces.
PCA is a linear coordinate system with a clear objective
Its strength comes from transparency: orthogonal directions, variance ordering, and low-rank reconstruction. Its limit is equally clear. Only linear variance gets ranked first, and a ranked direction invites a story.
The expensive version of that story is on the record. Maps of principal components of European gene frequencies, published by Cavalli-Sforza et al., were read for years as pictures of ancient migration events. Novembre and Stephens simulated data with no range expansion in the generating model and obtained the same pictures. Their 2008 report in Nature Genetics: “Here, we find that gradients and waves observed in Cavalli-Sforza et al.’s maps resemble sinusoidal mathematical artifacts that arise generally when PCA is applied to spatial data, implying that the patterns do not necessarily reflect specific migration events.” The cause sits in the algebra rather than in history. Spatial covariance matrices “have eigenvectors related to sinusoidal waves of increasing frequency”.
Use PCA to summarize, denoise, visualize, or preprocess when that objective matches the task. Do not promote a component into a causal latent factor without additional evidence. John Novembre is an author on both papers in this lesson. One read a map of Europe out of 0.30% of the variance. The other showed that such maps can be produced by a model containing none of the history they were credited with. Nothing in the PCA output distinguishes the two cases. Only the extra evidence does.
A principal component is mathematically defined and semantically provisional.
Key takeaways
- PCA projects centered data onto orthogonal directions of decreasing variance — in the POPRES analysis, PC1 and PC2 held about 0.30% and 0.15% of the variance and still reproduced the map of Europe to within 310 km for half the sample.
- Loadings define component directions while scores locate observations along them.
- SVD supplies the factorization used to compute PCA but has broader meaning: R's prcomp works “by a singular value decomposition of the (centered and possibly scaled) data matrix, not by using eigen on the covariance matrix”.
- Component signs are arbitrary and should not drive interpretation — a 2007 Sandia National Laboratories report documented the indeterminacy, and scikit-learn's svd_flip exists solely to impose a deterministic convention on it.
- Scaling changes feature influence and must be justified from semantics: the same scikit-learn Wine pipeline scores 35.19% unscaled and 96.30% with StandardScaler, and correlation-matrix PCs are a different set of components, not rescaled ones.
- High variance does not automatically mean high operational or causal importance — the leading PCs of the Mouse ENCODE data tracked sequencing flowcells, and Novembre and Stephens reproduced Cavalli-Sforza et al.'s migration gradients from a model with no range expansion in it.