Laws of Large Numbers, Central Limit Theorems, and Concentration

A poll's headline number is the mean of Bernoulli outcomes. The survey industry publishes what that mean is worth. The American Association for Public Opinion Research works the arithmetic on a simple random sample of 1,000 adults: “Consider a survey of 1,000 adults with an overall margin of sampling error of plus or minus 3 percentage points. If that sample includes, for example, 200 Hispanics, the overall results based on the subsample of Latinos is plus or minus 6.9 percentage points—the MOSE for a sample of 200.” Same survey, same file, same fieldwork. The margin is more than twice as wide. The guarantee attaches to the number of observations behind the estimate, not to the study that produced them.

The law of large numbers explains why an average can stabilize. The central limit theorem describes a rescaled distribution of estimation error. Concentration inequalities give finite-sample probability bounds. These results are related but not interchangeable. Each needs assumptions about independence, dependence, tails, moments, or boundedness. A careful analysis asks which conclusion is needed: consistency, an approximate sampling distribution, or a nonasymptotic guarantee.

Concentration has a date. Wassily Hoeffding published Probability Inequalities for Sums of Bounded Random Variables in March 1963, in the Journal of the American Statistical Association. The title states the scope exactly: sums, bounded summands, inequalities rather than limits. That is the whole difference between a concentration bound and a central limit theorem. One of the two says something about the sample size you actually have. The other describes a limit.

What follows is a set of places where the distinction was tested against data: a Gaussian approximation whose error is itself bounded and dated, 900,000 rows of real survey data in which dependence turned a 5 percent test into a 45 percent one, a bank describing daily moves a normal model puts beyond the age of the universe, a flu tracker whose errors predicted each other, and an audit whose guarantee is written into statute.

One sample mean appears in all three results, but the conclusions differ. A law of large numbers says the mean settles. That is a consistency statement: often asymptotic, sometimes surviving weak dependence, and silent about the shape of the remaining error. A central limit theorem gives that error a shape, and so supports approximate standard errors. The price is variance and regularity conditions, a convergence rate that can be slow, and tail events that may stay badly approximated. A concentration inequality bounds a deviation probability at the sample size in hand. It is nonasymptotic, depends on boundedness or tail assumptions, is often conservative, and is the right tool when a guarantee or a sample-complexity number is what the decision needs.

The middle claim is the one people treat as free. It is not free. The quality of the Gaussian approximation is itself a quantity, and it has been bounded and then sharpened for eighty years. In 1941 A. C. Berry proved that the maximum gap between the exact distribution of a standardised sum and the normal distribution is at most 1.88 times the third-absolute-moment ratio — an error of order 1/sqrt(n). His second theorem showed that no constant below 1/sqrt(2*pi) can work, so the result cannot be improved without limit. In 2011 Irina Shevtsova tightened the identically distributed case to Delta_n <= 0.3328(beta_3+0.429)/sqrt(n) and Delta_n <= 0.33554(beta_3+0.415)/sqrt(n), bringing the absolute Berry-Esseen constant to C_0 < 0.4748. Read those bounds as a practitioner. The approximation error falls at the same 1/sqrt(n) rate as the standard error. A distribution with a large third absolute moment sits in the numerator, paying for its own skew at every sample size.

A finite mean can exist while variance is infinite. In that case, a classical finite-variance CLT does not apply, and averages can remain volatile. Even with finite variance, rare large observations can make Gaussian approximations poor at practical sample sizes.

There is a well-documented instance of a normal model meeting its own tail. On 13 August 2007 Goldman Sachs was seeing "25-standard deviation moves, several days in a row". That was the firm's CFO, David Viniar, speaking to the Financial Times. A 2008 paper worked out what the sentence commits its speaker to. Under a normal distribution, a 25-sigma daily loss has probability 3.0570e-136 percent and an expected recurrence of once every 1.309e+135 years. Several days in a row of it is not an unlucky week. It is the distribution being wrong.

The Bank of England reached for the same episode when it explained why stress tests had failed. “To provide some context, assuming a normal distribution, a 7.26-sigma daily loss would be expected to occur once every 13.7 billion or so years. That is roughly the estimated age of the universe.” — Andrew Haldane, Bank of England, February 2009. Note what that example needs: not 25 sigma, only 7.26. The normal model already puts it past the age of the universe. Robust estimators or tail-aware bounds may be more appropriate. The word “large” in large-sample theory depends on the distribution, dependence, and error tolerance.

Match the theorem to the decision rather than citing a familiar result. The cleanest case of a decision that names its own guarantee is written into law. Colorado statute defines a risk-limiting audit as "an audit protocol that makes use of statistical methods and is designed to limit to acceptable levels the risk of certifying a preliminary election outcome that constitutes an incorrect outcome". The risk limit is the largest chance that the audit stops short of a full hand count when the reported outcome is wrong. A 5 percent risk limit therefore means at least a 95 percent chance that the audit leads to a full hand count correcting a wrong outcome. The number of ballots drawn follows from the risk limit and the contest margin. Not from a fixed percentage of ballots, and not from a limit as the number of ballots grows.

“Risk-limiting audits do not guarantee that the electoral outcome is right, but they have a large chance of correcting the outcome if it is wrong.” — Mark Lindeman and Philip Stark, 2012

That sentence is a finite-sample claim of exactly the kind a concentration argument supplies, and it is precise about what it does not claim. An asymptotic result could not have been written into a statute. The election happens once, on the ballots that exist.

The sequence is the same wherever the decision sits. Define the estimator: state the random quantity and its target. Audit dependence: identify groups, time structure, and adaptive data collection. Inspect tails: check bounds, moments, outliers, and rare-event behavior. Choose the claim: consistency, approximation, or finite-sample control. Quantify error: asymptotic variance, bootstrap, or a valid concentration bound. Validate empirically: simulate or resample under realistic dependence and sample sizes. That last move is the one that turned randomly generated placebo laws into a measured 45 percent rejection rate instead of an assumed 5 percent one.

Averages reduce independent variance by n, so their standard deviation shrinks by √n. One observation has variance σ². A sum of n observations has variance nσ² under independence. The average of n observations has variance σ²/n. The scaled error √n( X̄ − μ ) returns to variance σ², which is what allows a nondegenerate limit rather than a point mass.

That rate is a budget, and the people who buy samples for a living state the price. Doubling a sample from 1,000 to 2,000, AAPOR notes, "only reduces the margin of sampling error by about a single percentage point". A second thousand interviews buys one point. A narrow margin has to be bought at a scale most studies never reach. Pew Research Center's February - March 2025 survey of 5,123 respondents reports a margin of sampling error of plus or minus 1.6 percentage points, with sampling errors taking the effect of weighting into account. Five times the 1,000-adult sample, roughly half the margin. That is √n, priced.