Regression: Predicting Quantities Without Pretending to Know the Future

Regression predicts a quantity: time, demand, cost, temperature, probability-adjusted loss, remaining life. The numeric scale carries information about distance. An error of ten units is usually larger than an error of one.

Not every numbered label is a regression target. Postal codes, product identifiers and ordinal ratings may use digits without representing continuous magnitude.

Regression targets should be defined with the downstream action in mind.

Different regression outputs answer different operational questions. A point estimate, a set of conditional quantiles and a prediction interval are three different promises. Only the last two say anything about how often the model expects to be wrong.

A stated coverage level is a testable claim, and one supervisory regime has been auditing such claims by counting misses since 1996. The Basel Committee on Banking Supervision’s backtesting framework, published in January 1996, compares a bank’s 99th-percentile one-day value-at-risk against 250 days of trading outcomes. Section II states the principle: “The backtests to be applied compare whether the observed percentage of outcomes covered by the risk measure is consistent with a 99% level of confidence.” Then it counts. Zero to four exceptions in 250 days is the green zone; “The range from five to nine exceptions constitutes the yellow zone.”; ten or more is the red zone. Those boundaries are not conventions. They are the binomial arithmetic of the claim itself: “For 250 observations, it can be seen that five or fewer exceptions will be obtained 95.88% of the time when the true level of coverage is 99%.”

EU law wrote the same counts down. Article 366 of Regulation (EU) No 575/2013 defines an overshooting as “a one-day change in the portfolio’s value that exceeds the related one-day value-at-risk number generated by the institution’s model”. It then tabulates the capital multiplier addend by the number of overshootings in the most recent 250 business days: 0.00 for fewer than 5, then 0.40 at 5, 0.50 at 6, 0.65 at 7, 0.75 at 8, 0.85 at 9, and 1.00 at 10 or more. An interval that misses more often than it promised is not merely embarrassing. It is priced.

The loss does two things. It changes which summary of the target distribution is favored, and it sets how strongly large errors matter. Squared error pushes toward the conditional mean and lets a few extreme residuals dominate. Absolute error pushes toward the conditional median and survives heavy tails better. Quantile loss weights over- and underprediction asymmetrically, and returns a chosen percentile instead of a centre.

Quantile loss is not a classroom refinement, and its evaluation is not left to taste. The M5 “Uncertainty” competition ran on Kaggle from 2 March to 30 June 2020, on Walmart data, and demanded exactly this shape of output. Entrants had to submit “nine different quantiles (0.005, 0.025, 0.165, 0.250, 0.500, 0.750, 0.835, 0.975, and 0.995), that can sufficiently describe the complete distributions of future sales”. That line is from the abstract of the results paper in the International Journal of Forecasting. Nine numbers, not one, for each of 42,840 hierarchically aggregated series, scored by the weighted scaled pinball loss, across 909 submissions. That is what a service-level objective looks like when someone has to keep score on it.

Logarithms and other target transformations can stabilize scale or reduce the influence of extreme values. Predictions transformed back to original units may require care. Averages do not simply invert through nonlinear functions.

Document the transformed objective, the back-transformation, and the metric domain. Otherwise a mathematically valid pipeline can produce a misleading business interpretation.

The skew that motivates all of this is measurable, and US healthcare spending is where the health-economics literature measured it. An AHRQ statistical brief published in March 2025 reports: “In 2022, the top 1 percent of people ranked by their healthcare expenditures accounted for 21.7 percent of total healthcare expenditures, while the bottom 50 percent accounted for less than 3 percent.” That top 1 percent averaged $147,071 each. The same brief adds that “People with the top 5 percent of expenses in 2022 accounted for about half (49.7 percent) of healthcare expenses.” A mean fitted on the log of a distribution shaped like that, exponentiated back, is not the mean of the dollars.

The problem has a name and a fix that predate machine learning by decades. Naihua Duan proposed the smearing estimate in 1983, in the Journal of the American Statistical Association: a nonparametric estimate of the expected response on the untransformed scale, where the regression itself is fitted on a transformed scale. It is needed because exponentiating a fitted mean of log y does not return the mean of y. Manning and Mullahy reopened the question for skewed cost data in 2001, in the Journal of Health Economics. Which retransformation is correct, they showed, depends on whether the log-scale errors are heteroscedastic. If a pipeline fits on a log scale and reports in currency, the step back is a modelling assumption. Someone has to own it. It is not arithmetic.

A regression design review should expose scale and decision consequences. Define the units. Inspect the shape of the target. Match the loss to the decision. Request the uncertainty the action actually needs. Test extrapolation on time, geography and range slices. Audit the precision at which the output is communicated.

A ten-minute early estimate and a ten-minute late estimate have equal absolute size. Their operational consequences may differ. A missed upper tail can overflow capacity even when average error looks excellent.

Direction is the part an average hides. The most famous regression failure in public health is a failure of direction rather than of magnitude. Google Flu Trends was built to predict CDC influenza-like-illness rates, and its errors were almost all one way. Lazer and colleagues reported in Science on 14 March 2014: “GFT also missed by a very large margin in the 2011–2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011”. By February 2013 it was predicting more than double the CDC’s proportion of doctor visits for influenza-like illness. An independent evaluation in PLOS Computational Biology found the updated model overshot 2012–13 ILI surveillance by 268% nationally, 208% regionally and 296% locally. A mean absolute error, averaged over 108 weeks and reported as one number, would have shown a large error. It would not have shown that 100 of those weeks were high.

Report error by target range, direction, time horizon, and decision slice. Numeric performance becomes actionable only after the loss is connected to the cost of being wrong.