Maximum Likelihood, MAP, and Posterior Prediction

One likelihood and one prior can be turned into four different outputs, and they answer four different questions. The maximum-likelihood estimate is the parameter value maximizing the likelihood. The MAP estimate is the parameter value maximizing posterior density. A posterior summary is a mean, median, interval, or other function of p(θ|y). The posterior predictive is the distribution of future data after integrating over θ.

The fourth is not a more elaborate version of the first, and the difference has a date on it. In 1975 Aitchison compared two ways of predicting. One plugs an efficient point estimate into the model — the estimative density. The other mixes over the parameter — the predictive density. Under a Kullback-Leibler closeness criterion the plug-in loses. Everything else in this lesson is an instance of that gap. Where the point estimate stands in for the distribution, something measurable is given up.

Maximum likelihood chooses the parameter value that makes the observed data most probable or dense under the model. MAP chooses the parameter value with highest posterior density. A full Bayesian analysis retains the posterior distribution and averages predictions or decisions over it. Collapsing to one point can be convenient, but it discards uncertainty. The best summary depends on the decision loss. Posterior mean, median, mode, and predictive distribution solve different optimization problems.

Having a procedure and knowing what it means are separate achievements, and here they were ten years apart. Fisher published the numerical recipe in 1912, in On an absolute criterion for fitting frequency curves, written while he was a third-year undergraduate. The name arrived only in 1922, on page 323 of On the mathematical foundations of theoretical statistics, read to the Royal Society in November 1921. That is the same paper in which sufficiency and efficiency are set out. John Aldrich opens his 1997 history of those ten years with two sentences: “In 1922 R. A. Fisher introduced the method of maximum likelihood. He first presented the numerical procedure in 1912.”

The order is worth keeping. The procedure came first and the account of what it optimizes came after. That is still the order in which the confusion between an estimate and an inference tends to arrive.

The three methods share a likelihood. They differ in what they optimize and what they keep. Maximum likelihood uses only the likelihood for point estimation: asymptotically well studied under regularity, invariant under one-to-one reparameterization of the estimator, able to overfit weakly identified small datasets, and silent about prior information. MAP maximizes likelihood times prior density. It often resembles penalized optimization, depends on parameterization through density modes, returns one posterior mode, and can hide multimodality. Posterior prediction averages future outcomes over the posterior. It propagates parameter uncertainty, can improve decisions under small samples, requires integration or approximation, and still depends on model adequacy.

The third row is where the decision-theoretic verdict sits. Aitchison proved in 1975 that the estimative density — an efficient point estimate plugged into the model — is beaten by the predictive density that mixes over the parameter. His abstract says it in one line: “On a general criterion of closeness of fit based on a discriminating information measure the predictive method is shown to be preferable.” Later work restates that as a domination: under Kullback-Leibler risk the plug-in built on the MLE “is uniformly dominated” by the uniform-prior Bayes predictive density. On that account Aitchison's paper overturned the previous conventional wisdom. Integrating over the posterior is not a scruple attached to plugging in a point. It is the operation the plug-in loses to.

Start from the action and the loss rather than from a preferred estimator. State what action follows from the estimate or prediction. Specify the cost of overestimation, underestimation, and tail errors. Determine whether one point is sufficient for the stakes and the sample size. Compare MLE, MAP, posterior summaries, and predictive distributions against that loss. Then check calibration by testing repeated predictive behavior and decision outcomes.

That comparison has been run on real data twice, on the 1970 and the 2005 baseball seasons. Efron and Morris took the batting averages of 18 major league players after their first 45 official at-bats of the 1970 season and applied Stein's estimator. Their conclusions report the margin: “In the baseball, toxoplasmosis, and computer simulation examples, Stein's estimator and its generalizations increased efficiencies relative to the MLE by about 350 percent, 200 percent, and 100 percent.” Brown then repeated the exercise on all Major League players in the 2005 season. The naive predictor — the current average itself, which is the maximum-likelihood answer — came out the poorest performer in every situation. The observed average is the value that maximizes the likelihood of what was observed. It is not the value that best predicts what comes next.

Suppose one positive event appears in ten trials. The MLE is 0.1, and the whole question is how much less than a fact that number is. The neighbouring case, a count of zero, was answered in JAMA in 1983: “This "rule of three" states that if none of n patients shows the event about which we are concerned, we can be 95% confident that the chance of this event is at most three in n (ie, 3/n).” Hanley and Lippman-Hand's Table 2 gives the exact upper bound at n = 10 as 26%, against the rule's 30%. So a maximum-likelihood estimate of 0.0 from ten trials is compatible with a true rate of roughly one in four. That is the size of the thing a single number is being asked to stand for here.

Under a Gaussian observation model, an L2 penalty on coefficients can match a zero-mean Gaussian prior in the MAP objective. An L1 penalty corresponds to a Laplace prior under a compatible parameterization. This is not a later reinterpretation by Bayesians. Tibshirani derived it himself in the 1996 lasso paper, in a section headed “LASSO AS BAYES ESTIMATE”, where the lasso appears as the Bayes posterior mode under independent double-exponential priors and the normal density is named as ridge regression's implicit prior.

What the derivation delivers is an equivalence of modes and nothing wider, and the Bayesian authors say so about their own construction. Park and Casella write, in The Bayesian Lasso: “Maximising the posterior, though sometimes convenient, is not a particularly natural Bayesian way to obtain point estimates. For instance, the posterior mode is not necessarily preserved under marginalisation.” They then show the divergence on data. On the diabetes data of Efron et al. (n = 442, p = 10), fully Bayesian posterior medians computed from 10,000 Gibbs iterations after 1,000 burn-in trace a different path from the lasso estimates.

The posterior normalization, uncertainty, parameterization, and predictive integration all remain different. Reading a penalty as a prior is a modeling claim. Someone should check it.