PCA in Practice: Scaling, Whitening, and Reconstruction

Whitening changes the geometry a downstream model sees. It strips the retained components of their original variance scale. scikit-learn states the trade in one sentence, in the description of PCA's `whiten` parameter: “Whitening will remove some information from the transformed signal (the relative variance scales of the components) but can sometime improve the predictive accuracy of the downstream estimators by making their data respect some hard-wired assumptions.” The same page defines the operation exactly. Multiply the components by sqrt(n_samples), divide by the singular values, and every component has unit variance. The cost is arithmetic, not opinion.

Which estimators collect the gain has been measured. Coates and colleagues made whitening an explicit axis of variation on CIFAR-10 in 2011, across four unsupervised feature learners: sparse auto-encoders, sparse RBMs, K-means and Gaussian mixtures. Their “Effect of whitening” section is blunt about where it matters: “For the clustering algorithms, however, we see that whitening is a crucial pre-process since the clustering algorithms cannot handle the correlations in the data”. Their best configuration used whitening, a 1-pixel stride and a 6-pixel receptive field. It reached 79.6% on CIFAR-10 and 97.2% on NORB.

So the three columns below are not three tastes. Ordinary scores keep the fitted variance magnitudes, which is what reconstruction needs. Whitened scores equalize the retained components, which is what a distance-based or clustering estimator may need — at the price the library names. Keeping the prepared original features avoids component interpretation altogether. That is sometimes the right call when the signal is sparse or rare.

Shipping a PCA model takes more than a matrix of components. Five pieces of state travel together. Lose one and future transforms go silently wrong rather than loudly broken.

The preprocessing state is the imputer, the transformer, the centering means and any scaling parameters. The component matrix holds the retained feature-space directions, with a fixed sign convention for reporting. Sign is arbitrary, and flipping it between runs breaks every downstream comparison. The variance information is the singular values, the explained variances and the cumulative ratios from the fit data — descriptive of that dataset and of no other. The projection contract is the rule for transforming and inverse-transforming new observations. The monitoring profile is the score distributions, the reconstruction error, the missingness and the support diagnostics you need in order to notice drift.

In 2014 a Mouse ENCODE study compared human and mouse tissues, and the samples grouped by species rather than by tissue: “the expression for many sets of genes was found to be more similar in different tissues within the same species than between species”. That was the headline of Lin and colleagues. Read as biology, it is a striking claim about how far mouse and human tissue programs have diverged.

A reanalysis of the same data followed in 2015. The assignment of samples to sequencing flowcells and lanes was nearly completely confounded with species. The dominant axis of variation was the instrument, not the organism. Gilad and Mizrahi-Man state the result in their abstract: “When we account for the batch effect, the corrected comparative gene expression data from human and mouse tend to cluster by tissue, not by species.”

Nothing in a variance ratio can tell you which of those two stories you are looking at. Explained variance is a property of the reference dataset and of how it was collected. It does not know which variation the downstream task values. It does not know which tail event must survive a truncation. And it does not know that the biggest direction in the matrix belongs to the machine that produced it.

Use several candidate ranks and evaluate the relationships the pipeline must preserve. Tooling exists for exactly this boundary. The scikit-learn user guide, in the Safety bullet of “Pipelines and composite estimators”, says that “Pipelines help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors”. The guarantee is mechanical — same samples, transformers and predictors alike. That is why the boundary is better enforced by a pipeline object than by discipline.

Five steps carry the work. First, fit within boundaries: learn all preprocessing and PCA parameters on the reference training period or fold. Second, compare ranks: review variance, reconstruction, neighbor preservation, runtime and downstream performance together, rather than one ratio alone. Third, test rare slices: measure retained signal for minority groups, anomalies and low-variance operational events, since a truncation discards those directions first. Fourth, evaluate whitening: compare ordinary and whitened scores for the specific downstream estimator. Coates and colleagues found the effect decisive for clustering methods and not uniform across the four learners they tested. Fifth, monitor future data: track score distributions, reconstruction error, missingness and feature drift.

Several diagnostics should be reviewed together — and the rank does not have to be chosen by eye at all. In 2000 Minka treated PCA as density estimation and picked the dimensionality by Bayesian model selection, using Laplace's method. His abstract claims a great deal: “In simulations, it is convincingly better than cross-validation and other proposed algorithms, plus it runs much faster.” The method is not a curiosity in a proceedings volume. It ships. The scikit-learn API reference for PCA records that “If n_components == 'mle' and svd_solver == 'full', Minka's MLE is used to guess the dimension”, and cites Minka for it.

Fewer components improve compression and can reduce noise. More components preserve detail and instability. The correct balance depends on the downstream relation and the failure cost. Where the noise model is known, though, the balance can be computed rather than negotiated. Gavish and Donoho derived the AMSE-optimal hard threshold for singular values in 2014. For a square n-by-n matrix in white noise of known level σ, the cutoff is (4/√3)·√n·σ ≈ 2.309·√n·σ. When σ is unknown it becomes 2.858·y_med, where y_med is the median empirical singular value.

The threshold has a textbook demonstration. Brunton and Kutz describe it in Data-Driven Science and Engineering: “A recent theoretical breakthrough determines the optimal hard threshold τ for singular value truncation under the assumption that a matrix has a low-rank structure contaminated with Gaussian white noise.” Then they run it on a synthetic rank-2 matrix with n = 600 and σ = 1. The optimal cutoff, τ ≈ 56.6, keeps exactly the two true components. The 90% cumulative-energy criterion plotted in the same figure sits at r = 403. Two components of signal, four hundred and three components kept. That is what a ceremonial percentage does to a matrix whose true rank is small and whose noise is large.

Store the full fit state and the evidence used to choose rank and whitening. A cumulative variance percentage alone is not a deployment rationale.