Skip to content
AI.info

Mathematical foundations

Expectation, Variance, Covariance, and Gaussian Geometry

Understand expected values, variance, covariance matrices, correlation, multivariate Gaussians, and the limits of moment-based reasoning.

By the end you can

Comparison

Covariance and correlation answer related but distinct questions

Scaling can change one while leaving the other fixed. Covariance is carried in the product of the original units, so rescaling a feature rescales it. Correlation divides by both standard deviations, lands between −1 and 1, and is therefore comparable across pairs and blind to units.

Neither is a licence to read the number as a description of the relationship. One published dataset has an overall Pearson correlation of +0.81 while every subgroup inside it is individually negative. Matejka and Fitzmaurice built it in 2017, by simulated annealing — Simpson's paradox produced to order. A single correlation coefficient reported for that dataset is arithmetically correct and points the wrong way for every group the dataset is made of. Independence is a stronger claim than either statistic can settle. It concerns the whole joint distribution, and a correlation matrix is not evidence for it.

FigureComparison · 3 columns

Covariance

Co-movement in the product of original units.

  • Sensitive to feature scale
  • Sign indicates linear direction
  • Magnitude depends on units
  • Forms the covariance matrix

Correlation

Standardized covariance between −1 and 1.

  • Dimensionless
  • Comparable across pairs
  • Undefined when a variable has zero variance
  • Still captures only linear association

Independence

Full factorization of the joint distribution.

  • Implies zero covariance when moments exist
  • Not implied by zero correlation in general
  • Concerns all distributional structure
  • Needs stronger evidence than a correlation matrix

Moments compress a distribution into selected features

Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean. Covariance measures how two centered variables move together. These summaries are powerful because they support algebraic calculations and diagnostics. They are incomplete because many distinct distributions share the same moments.

A useful habit is to ask what a moment captures and what it discards. Mean and variance can hide asymmetry, multimodality, heavy tails, and rare catastrophic outcomes. The rest of this lesson asks that question of published records rather than of hypotheticals: four datasets that share a regression line, thirteen that share five statistics to two decimal places, a correlation matrix that is not a correlation matrix, a forest whose error is capped by the correlation between its trees, an ensemble of 57 climate models carrying the information of 11, and a risk number that halved because a spreadsheet divided by the wrong quantity.

A moment is a summary of a distribution, not a substitute for the distribution.

Case

Anscombe's four datasets, and the 2017 method that manufactures them

Four datasets, eleven points each, and one summary table between them. Anscombe published them in February 1973, in The American Statistician. All four share a mean x of 9 and a sample variance of x of 11, both exact. All four share a mean y of 7.50, a correlation of about 0.82 — 0.816 to three decimal places — and the same least-squares regression line, y = 3.00 + 0.500x. Plotted, they look nothing alike.

The demonstration has since become a procedure. “Datasets which are identical over a number of statistical properties, yet produce dissimilar graphs, are frequently used to illustrate the importance of graphical representations when exploring data,” wrote Justin Matejka and George Fitzmaurice in 2017, opening the abstract of a paper on generating exactly such datasets by simulated annealing. Their method produced the Datasaurus Dozen: 13 datasets that agree to two decimal places on mean x = 54.26, mean y = 47.83, sd x = 16.76, sd y = 26.93 and Pearson r = −0.06. One of them plots as a dinosaur. The datasets ship publicly in the R package datasauRus, whose own recomputation returns mean x 54.3, mean y 47.8, sd x 16.8, sd y 26.9 and correlations between −0.060 and −0.069 across all 13.

So agreement in the summary table does not establish that two datasets resemble each other. It is now something a search procedure can be asked to produce on demand.

Linearity of expectation does not require independence

For constants a and b, E[aX+bY]=aE[X]+bE[Y] whenever expectations exist. Dependence between X and Y does not change this identity. Variance behaves differently: Var(X+Y)=Var(X)+Var(Y)+2Cov(X,Y). The covariance term disappears only under zero covariance. Keeping those two rules apart is essential in ensembles, portfolios, averaged measurements, and error propagation.

The asymmetry is the whole reason averaging is not free. A mean of many predictors inherits their expectations linearly. It inherits their variances only through the full covariance structure. That structure is the ceiling the ensemble section below puts published numbers on.

Visual

First and second moments

These concepts extend from scalars to random vectors, and the extension arrives with a constraint attached. A covariance matrix must be symmetric positive semidefinite, and a correlation matrix must be that with a unit diagonal. Positive semidefiniteness is not a decoration on the object. It is a condition that matrices assembled from real data violate.

It breaks first in finance, in matrices of correlations between stocks. “Given a symmetric matrix, what is the nearest correlation matrix—that is, the nearest symmetric positive semidefinite matrix with unit diagonal? This problem arises in the finance industry, where the correlations are between stocks.” That is Nicholas Higham, opening his abstract in the IMA Journal of Numerical Analysis in July 2002. In that finance application, he notes, “the original matrix has many zero or negative eigenvalues” — the table of pairwise numbers does not correspond to any joint distribution at all. Higham formalises the repair as a projection onto the cone of symmetric positive semidefinite matrices with unit diagonal, and solves it by modified alternating projections. Qi and Sun gave a quadratically convergent Newton method for the same problem in 2006. A whole numerical literature exists for one reason. The geometry of second moments is a genuine constraint, and estimated matrices break it.

FigureHierarchy · 5 levels

  • Mean E[X]

    Location or average under the probability model.

    • Variance Var(X)

      Average squared deviation from the mean.

      • Covariance Cov(X,Y)

        Linear co-movement after centering.

        • Covariance matrix Σ

          All pairwise second moments of a random vector.

          • Correlation

            Dimensionless covariance scaled by standard deviations.

Second moments describe spread and linear geometry around the mean.

Example

Why correlated errors limit averaging gains

Suppose n predictors have equal error variance σ² and pairwise correlation ρ. What the ensemble buys you depends on ρ, not on n alone. Leo Breiman proved the corresponding statement for forests and put it in his first sentence: “The generalization error of a forest of tree classifiers depends on the strength of the individual trees in the forest and the correlation between them,” from the abstract of Random Forests in 2001.

  • Average error: The ensemble mean has variance σ²[ρ+(1−ρ)/n] — a correlation floor plus a term that shrinks with n.
  • Independent case: If ρ=0, variance falls like 1/n. Breiman's Theorem 2.3 bounds generalization error by ρ̄(1−s²)/s², where ρ̄ is the mean correlation between members and s their strength.
  • Correlated case: As n grows, variance approaches ρσ² rather than zero. Theorem 11.2 gives PE*(forest) ≤ ρ̄·PE*(tree) for regression, so the forest “decreases the average error of the trees employed by the factor ρ̄”.
  • In numbers: On Boston Housing, Breiman's out-of-bag estimates are PE*(tree) = 26.3 with correlation .45, and the forest's test-set mean-squared error is 10.2 — inside the 0.45 × 26.3 ceiling.
  • Design consequence: Ensemble diversity matters mathematically through error covariance, not through model count alone.

Analogy

A shared tide is what covariance measures

Boats in a harbor each move individually, but a shared tide makes them rise and fall together. Variance measures each boat's motion; covariance captures shared movement. Subtracting the mean water level isolates deviations around the common baseline.

Multi-model climate ensembles are that shared tide made measurable. A 2023 paper in npj Climate and Atmospheric Science computed an effective number of independent models, M_eff, of 10.7 for CMIP3, 11 for CMIP5 and 11 for CMIP6. Measured inter-model similarity had meanwhile risen from 55% across 24 CMIP3 models to about 76% across 47 CMIP5 models and 80% across 57 CMIP6 models. “After accounting for similarity, the information from 57 CMIP6, 47 CMIP5, and 24 CMIP3 models can be explained by just 11 independent models without significant differences in globally averaged climate change statistics,” the abstract says. Fifty-seven boats, eleven tides. A separate 2020 paper arrives at the same structure from another direction, documenting that CMIP models share code and components and quantifying the effect of weighting 33 CMIP6 models for interdependence.

Nonlinear dependence escapes this picture entirely. Two variables can be tightly related while having zero covariance, in the way boats could follow a symmetric nonlinear pattern.

Covariance records linear co-movement around the mean, not every form of dependence.

Steps

Using moments without losing the distribution

Treat moments as part of a diagnostic set. Treat the skipping of that set as a live risk rather than a style preference.

A new Value-at-Risk model took effect at JPMorgan Chase's Chief Investment Office on 27 January 2012. It “immediately reduced the CIO's VaR by 50%, from $132 million to $66 million”, in the words of the U.S. Senate Permanent Subcommittee on Investigations. The bank's own Management Task Force later found how. “Specifically, after subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended,” its report of 16 January 2013 records — an error that “likely had the effect of muting volatility by a factor of two and of lowering the VaR”. The Subcommittee's staff report describes the manual spreadsheets on which VaR was computed as “error prone”. The Synthetic Credit Portfolio's losses “reached at least $6.2 billion” in 2012.

A halving of the reported number on the day a model changed is what step four is meant to catch. A hand-built covariance computation is what step two is meant to inspect. The routine below is cheap. Not running it was not.

FigureProcess · 5 steps

  1. 1. Compute location and scale

    Estimate means, medians, variances, and robust alternatives.

  2. 2. Inspect dependence

    Compare covariance, correlation, and conditional plots.

  3. 3. Examine shape

    Check tails, skew, multimodality, and bounded support.

  4. 4. Test stability

    Recompute moments across time, slices, and resamples.

  5. 5. Match decisions

    Use tail or quantile summaries when average behavior is insufficient.

Key idea

The multivariate Gaussian is unusually determined by two moments

A multivariate Gaussian distribution is fully specified by its mean vector and covariance matrix. Its constant-density surfaces are ellipsoids shaped by covariance, and for jointly Gaussian variables, zero covariance implies independence.

This implication does not extend to arbitrary distributions. Gaussian models are mathematically convenient and often useful, but tails, bounded support, multimodality, and skewness can make them inappropriate.

Bank regulators have acted on precisely that gap, demoting the single quantile in favour of a tail average. The Basel Committee on Banking Supervision's standard of 14 January 2016 on minimum capital requirements for market risk lists among its key enhancements “A shift from Value-at-Risk (VaR) to an Expected Shortfall (ES) measure of risk under stress”. It then fixes the level: “In calculating the expected shortfall, a 97.5th percentile, one-tailed confidence level is to be used.” The US banking agencies proposed the same move in the Federal Register on 18 September 2023, in a rule that “would replace the VaR-based measure of market risk with an expected shortfall-based measure that better accounts for extreme losses”, defining expected shortfall as the average of all potential losses exceeding VaR. The change is an admission in regulation that the summary you report is a modelling assumption about shape.

Gaussian conclusions should not be exported to non-Gaussian data without checking the distributional assumption.

Position

A mean, a standard deviation and a correlation smuggle in an assumption about shape

Mean, standard deviation, correlation. A dataset arrives with that short table attached, and the table is easy to mistake for a description of the data. Anscombe settled what it is worth in February 1973. Four datasets, eleven points each, sharing a mean x of 9, a mean y of 7.50, a correlation of about 0.82 and the least-squares line y = 3 + 0.5x — and looking nothing alike when plotted. Matejka and Fitzmaurice turned that demonstration into a search procedure in 2017. Simulated annealing gave them the 13 datasets of the Datasaurus Dozen, held to two decimal places at mean x 54.26, mean y 47.83, sd x 16.76, sd y 26.93 and r = −0.06, one of them a dinosaur. The same paper built a dataset whose overall Pearson correlation is +0.81 while every subgroup in it is negative. Agreement in that table does not establish that two datasets resemble each other. It is something a procedure can now be asked to manufacture.

The table feels sufficient because there is a much-used family for which it is. A multivariate Gaussian is fully specified by its mean vector and covariance matrix, and for jointly Gaussian variables zero covariance implies independence. Both statements are true. Neither travels. Elsewhere, correlation still records only linear association, zero correlation need not deliver independence, and many distinct distributions share the same moments. Even the matrix itself is less docile than it looks. Positive semidefiniteness is a constraint that correlation matrices built from real data violate, which is why Higham had to pose the nearest-correlation-matrix problem in July 2002, and Qi and Sun to give a quadratically convergent method for it in 2006. A report that stops at means and covariances has assumed a shape without saying so. Asymmetry, multimodality, heavy tails and rare catastrophic outcomes are what it assumed away.

This is not an argument against moments. They carry the algebra that makes error propagation and ensembles tractable, and second moments are frequently where the real answer lives. For n predictors with equal error variance σ² and pairwise correlation ρ, the ensemble mean has variance σ²[ρ+(1−ρ)/n]. Breiman's Theorem 11.2 states the same ceiling for forests, PE*(forest) ≤ ρ̄·PE*(tree). The climate ensembles measure it: 57 CMIP6 models carry about as much independent information as 11. The argument is against the summary standing in for the distribution it summarizes. When it does, the cost is not academic. The CIO's new model cut reported VaR from $132 million to $66 million on 27 January 2012, the spreadsheet beneath it divided by a sum instead of an average, and the Synthetic Credit Portfolio's losses “reached at least $6.2 billion”. Basel's answer in 2016, and the US banking agencies' proposal of 18 September 2023, was to stop reporting that single number and report a tail average instead, at “a one-tail, 97.5th percentile confidence interval”. The lesson's own routine says the same thing at smaller scale: compare conditional plots, examine tails, skew, multimodality and bounded support, recompute across time, slices and resamples, and reach for quantile summaries when average behavior is not what the decision turns on.

A summary table agrees far more easily than the distributions behind it do.

Key takeaways