Point Estimation: Bias, Variance, Consistency, and Efficiency

A parameter is a feature of a probability model or population. An estimator is a function of sampled data. An estimate is the realized numerical result. Because the sample is random, the estimator has a sampling distribution, and bias, variance, consistency, and efficiency describe that rule across repeated datasets rather than one observed number.

The estimand must be defined before the estimator can be judged. Google Flu Trends is the dated, measured case of what happens when it is not. In February 2013 the model was running far ahead of the disease it was supposed to track. Four researchers set the failure out in Science on 14 March 2014: “Nature reported that GFT was predicting more than double the proportion of doctor visits for influenza-like illness (ILI) than the Centers for Disease Control and Prevention (CDC), which bases its estimates on surveillance reports from laboratories across the United States”. An independent reassessment then sized the overshoot across the whole 2012/13 season. It put the model's estimates of epidemic intensity “far greater than ILI surveillance data at the national (268%), regional (208%) and local (296%) levels”.

The fitting procedure explains the result. The Science piece describes it in one line: “the methodology was to find the best matches among 50 million search terms to fit 1152 data points”. A rule selected that way can be very precise about something. What it was precise about was search behaviour. The estimand was doctor visits.

These criteria describe different aspects of behavior under repeated sampling, and the first two are not this course's private vocabulary. The International Vocabulary of Metrology (JCGM 200:2012) defines measurement bias at clause 2.18 as “estimate of a systematic measurement error”. It keeps that separate from random measurement error, at clause 2.19, whose replicate values “form a distribution that can be summarized by its expectation, which is generally assumed to be zero, and its variance”. Systematic measurement error is clause 2.17. The NIST/SEMATECH e-Handbook of Statistical Methods makes the same split quantitatively, in a section called “Bias and Accuracy”: “Bias is a quantitative term describing the difference between the average of measurements made on the same object and its true value.”

Two standards bodies, one distinction. An offset in the expectation on one side, spread across repetitions on the other. Consistency, efficiency and robustness then ask what happens to that pair as data grow, as alternatives compete, and as the assumed model is violated.

A consistent estimator converges to its target as sample size grows under specified assumptions. It can still perform poorly at realistic sample sizes. An inconsistent estimator can occasionally look better in a small experiment. Finite-sample risk, computation, and robustness remain relevant. The mode of convergence also matters: convergence in probability, almost sure convergence, and convergence in distribution are not interchangeable.

Consistency can also fail outright, and the canonical case is named and dated. In 1948 Neyman and Scott took X_ij independent N(mu_j, sigma^2) for i = 1, 2 and j = 1, ..., n. That is two observations on each of n groups, every group carrying its own mean. Each new group brings a new parameter along with its data, so the information never concentrates. Stigler states the outcome plainly: “the maximum likelihood estimate of sigma^2 consistently estimates half the correct value (Neyman and Scott, 1948)”. The bias E(sigma-hat^2) - sigma^2 = -sigma^2/2 does not shrink as n grows. No amount of additional data repairs it.

The result has a modern restatement in general form. Dhaene and Jochmans write that “Neyman and Scott (1948) showed that psi-hat is not, in general, a consistent estimator of the true value of psi as n → ∞ while m remains fixed”, with the pooled within-group estimate converging to (1 − 1/m)·psi for m observations per group. At m = 2 that limit is exactly half the truth. That is the 1948 example again. And Stigler explains why this is not an antiquarian point: “The Wald–Neyman–Scott example was of more practical import, and still serves as a warning of what might occur in modern highly parameterized problems, where the information in the data may be spread too thinly to achieve asymptotic consistency.”

Location estimators trade efficiency under a model against resistance to outliers, and that trade is settled with measured risk rather than taste. In a 1997 NBER working paper, Bryan and colleagues searched for the trim that estimates core inflation best. The abstract gives the answer: “We find that trimming 9% from each tail of the CPI price-change distribution, or 45% from the tails of the PPI price-change distribution, yields an efficient estimator of core inflation for these two series, although lesser trims also produce substantial efficiency gains.” Their optimal trimmed estimators came out “nearly 23% more efficient (in terms of root-mean-square error) than the standard mean CPI, and 45% more efficient than the mean PPI”.

A central bank ships both alternatives every month. From the BLS monthly CPI report, the Federal Reserve Bank of Cleveland computes a median CPI — “the one-month inflation rate of the component whose expenditure weight is in the 50th percentile of price changes” — and a 16 percent trimmed-mean CPI, “a weighted average of one-month inflation rates of components whose expenditure weights fall below the 92nd percentile and above the 8th percentile of price changes”. Mean, median and trimmed mean are three published numbers computed from the same prices. The trimming fraction is part of the estimator's definition, not a cosmetic adjustment to it.

Comparing point estimators is worth doing in this order, and the U.S. Census Bureau runs steps 4 and 6 against its own count in public. The 2020 Post-Enumeration Survey treats the census as an estimator and measures its bias. The Bureau's news release of 10 March 2022 gives the headline number: “It estimated a net coverage error of -0.24% (or 782,000 people) with a standard error of 0.25% for the nation, which was not statistically different from zero.” On the aggregate, then, no detectable bias.

Step 6 is where the audit earns its keep. The same survey found statistically significant undercounts of 3.30% for the Black or African American alone-or-in-combination population, 4.99% for the Hispanic or Latino population, and 5.64% for American Indian or Alaska Native people living on reservations. It found overcounts as well: 1.64% for non-Hispanic White alone, 2.62% for Asian alone-or-in-combination. The U.S. Government Accountability Office reports the same survey from outside, and reports both halves of it: “the survey estimated no statistically significant net coverage error for the national population count”, while “two geographic regions and 14 states had statistically significant net coverage errors”.

A national figure that averages to nothing is composed of signed, significant errors in exactly the subpopulations the count is used to fund and apportion. Aggregate unbiasedness is a property of one estimand. It does not transfer to the estimands built out of it.

Early in the 1970 season, 18 major-league players had exactly 45 official at-bats each. Efron and Morris used each player's average over those 45 at-bats to predict his average over the rest of the season. Clemente had gone 18 for 45, an observed 0.400. Munson 8 for 45, an observed 0.178. The grand average of the 18 was 0.265. The James–Stein rule shrinks each individual average toward that grand average, which is a systematic bias by construction. It beat the player's own observed average for 16 of the 18. Writing in Scientific American in 1977, the two of them gave the totals: “The observed averages y have a total squared error of .077, whereas the squared error of the James-Stein estimators is only .022. By this comparison, then, Stein's method is 3.5 times as accurate.”

The margin has been restated since. Efron reports that the James–Stein “total squared prediction error was less than one-third of that for the observed averages”. Brown describes the same experiment independently: “They used averages from the first 45 at-bats of a sample of 18 players in 1970 in order to predict their batting average for the remainder of the season”. Nothing about the biased rule is unbiased, and nothing about the unbiased rule was more accurate.