Regression Metrics and Residual Diagnostics

On 2 November 2021 Zillow Group shut down Zillow Offers, the arm that bought houses on the strength of its own price predictions. The third-quarter results carried a roughly $304 million inventory write-down. The company told investors to expect a further $240-265 million of losses in the fourth quarter. It cut about 25% of its workforce — around 1,600 jobs out of 6,400.

The reason its co-founder and CEO gave was not that the model was wrong on average. It was the width of the error distribution. “We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility,” Rich Barton said the same day. The typical error was survivable. The tail was not.

A model that is modestly wrong about every ordinary house can post the same root mean squared error as one that is nearly perfect on most and catastrophically wrong on a few. Only one of those two puts unsellable inventory on a balance sheet. Regression evaluation therefore begins with the error distribution, not with one average. Magnitude, direction, tail behavior, target scale, and context all matter.

Which average to report is itself contested. In 2014 Chai and Draxler answered a proposal to drop RMSE altogether. They state the charge first: “Willmott and Matsuura (2005) have suggested that the RMSE is not a good indicator of average model performance and might be a misleading indicator of average error”. Their reply is short: “the proposed avoidance of RMSE in favor of MAE is not the solution”. RMSE, they hold, “is more appropriate to represent model performance than the MAE when the error distribution is expected to be Gaussian”. And their conclusion asks for more than one number: “a combination of metrics … are often required to assess model performance”. Nobody wins the argument about the single best average. That is the finding.

Each metric weights errors differently, and the choice is not academic. One of them once settled a million dollars. Netflix ranked its public contest on a single squared-error average, and its organisers, Bennett and Lanning, fixed the number to beat: “The company reported the RMSE performance of Cinematch trained on the Prize dataset against the quiz subset as 0.9514, a 9.6% improvement over simply predicting individual movie averages.” The Grand Prize of $1,000,000 went to whoever improved on that accuracy by a further 10%. The data was over 100 million ratings from 480,189 users on 17,770 movies.

Two details of that design matter more than the algorithms it produced. Entrants were told only their quiz-subset RMSE. The test-subset RMSE that actually decided the prize was never disclosed to them. And the whole ranking rested on four decimal places of one squared-error average. By construction that summary weights a handful of badly predicted ratings far more heavily than a mass of nearly correct ones. It says nothing about which users or which films the misses fell on.

The four families below make that weighting choice explicit: absolute error in the target's units, squared error and its rooted form, the mean-baseline comparison, and errors divided by a reference magnitude.

Errors for four cases are 1, 1, 1, and 9 units. Nothing about the four numbers changes; only the summary applied to them does. Each summary reports a different lesson about the same four cases.

Structure in residuals suggests missing information or a mismatched objective. Google Flu Trends is the documented case. It ran high in 100 of the 108 weeks from 21 August 2011 to 1 September 2013, and it overshot the 2011-2012 level by more than 50%. Science published that count on 14 March 2014. Missing high in 100 of 108 weeks is not bad luck. It is a pattern any residual plot against time would have shown.

The authors say so directly: “These errors are not randomly distributed. For example, last week's errors predict this week's errors (temporal auto-correlation), and the direction and magnitude of error varies with the time of year (seasonality).”

A baseline comparison made the same point in one line of arithmetic. Out of sample, Google Flu Trends recorded a mean absolute error of 0.486. A lagged-CDC baseline recorded 0.311. A combined GFT-plus-CDC model recorded 0.232. A model built from nothing but week-old official figures beat the search data outright.

An independent team measured the overestimation from the other side. It put GFT epidemic-intensity estimates far above ILI surveillance in 2012/2013: nationally 268%, regionally 208%, locally 296%.

The patterns worth looking for are the ones that case displays: level bias, error spread that grows with the target, curvature, runs and cycles in time, and large errors concentrated in one region, product, device, or severity group.

MAPE and related measures divide by a reference magnitude. That makes zeros and near-zeros unstable, and gives small targets disproportionate weight. Symmetric variants change those choices but do not erase them.

Regulators have had to write the policy down in full. FDA's final guidance of September 2020 on self-monitoring blood glucose test systems for over-the-counter use sets the criterion in one sentence: “Your studies should demonstrate that your SMBG is sufficient for this purpose by showing that 95% of all SMBG results in this study are within +/- 15% of the comparator results across the entire claimed measuring range of the device and that 99% of all SMBG results are within +/- 20% of the comparator results across the entire claimed measuring range of the device.”

The interesting part is the remedy. At low glucose the denominator gets small and the percentage explodes, and some devices cannot hold 15% there. The agency does not soften the metric. It tells the manufacturer to raise the bottom of the claimed measuring range — to stop making claims where the denominator cannot support them.

ISO 15197:2013 solves the same problem the other way, by switching denominator: at least 95% of results within +/-15 mg/dL below 100 mg/dL, and within +/-15% at or above it. Below 100 mg/dL the percentage is abandoned outright and an absolute tolerance takes over.

Use domain-aware denominators, scale by a meaningful baseline, or report separate ranges. Always state exclusions and clipping rules.

Use several views that complement rather than duplicate one another. Report error in units stakeholders understand. Show the distribution with median, upper quantiles, tail counts and signed bias. Plot residuals against prediction, target, time and important features. Compare a baseline — mean, median, seasonal naive, or the incumbent — under the same sample. And slice consequences where thresholds, capacity or safety change.

That last step rewrote American clinical practice once. A creatinine equation was refit with age and sex and without race, and Inker and colleagues reported the validation in the New England Journal of Medicine in 2021. Pooled accuracy for the two groups looked much alike: P30 of 87.2% in Black participants against 86.5% in non-Black participants. A report that stopped there would have shown nothing.

The signed residual, measured per group, did not agree: “The new eGFRcr(AS) equation underestimated measured GFR in Black participants (3.6 ml per minute per 1.73 m2; 95% CI, 1.8 to 5.5) and overestimated measured GFR in non-Black participants (3.9 ml per minute per 1.73 m2; 95% CI, 3.4 to 4.4).” One group's predictions run low and the other's run high. Any average over both cancels the two biases against each other and reports a model in balance.

On 23 September 2021 the NKF-ASN Task Force recommended “the adoption of the new eGFR 2021 CKD EPI creatinine equation that estimates kidney function without a race variable”. Its earlier interim statement had already held that “Race modifiers should not be included in equations used to estimate kidney function”. Here the number decides who is referred and who is transplanted. A slice-level residual at that threshold moved two national societies.