Loss Families and Output Contracts

The correct loss depends on what one prediction is supposed to mean.

The six contracts below are not stylistic preferences. Each one appears further down as a measured system. Fourteen co-occurring observations labelled on 224,316 chest radiographs. Nine named quantiles scored on 42,840 Walmart time series. A search leaderboard with 1,055 registered teams, where the winners were ranked on ERR rather than the NDCG the leaderboard displayed. A translation model that assigns its best score to the empty string. In each case the choice of contract decided the outcome, not the arithmetic.

Each pipeline can run correctly while teaching the wrong behavior. The cases below are documented ones, with the figures their own authors reported.

Squared error corresponds to a particular view of residuals when interpreted probabilistically. Absolute error emphasizes median behavior, while quantile loss targets a chosen conditional quantile.

Cross-entropy compares a target distribution with predicted probabilities, yet real labels may be uncertain, censored, or incomplete. Selecting a loss therefore commits the project to a claim about what the target means.

Label smoothing is that claim written down. It replaces the one-hot target with a mixture of the ground-truth label and a fixed prior, keeping most of the mass on the true class and spreading the rest. The Inception architecture paper named the trade in 2015 and priced it: “In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and ϵ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error”. The algebra of cross-entropy did not change. Only the assumed target did.

The same kind of commitment has a 1964 original. Huber's estimator uses a quadratic loss for small deviations and a linear one for large. PyTorch ships it as torch.nn.HuberLoss(reduction='mean', delta=1.0), which “uses a squared term if the absolute element-wise error falls below delta and a delta-scaled L1 term otherwise”. Setting that delta is not a numerical detail. It states how much a project will let outliers steer the fit.

This map connects a loss choice to the behavior it rewards. Squared error gives large residuals rapidly increasing influence. Absolute error counts magnitude linearly. Huber loss is quadratic below its delta and linear above it. Quantile loss uses asymmetric penalties to target a chosen conditional quantile. Distributional objectives predict parameters, or a full conditional distribution.

The Zillow Offers wind-down is what the first row costs when the residual tail is the business risk: roughly a $304 million inventory write-down in Q3 2021, $240–265 million of further losses expected in Q4, about 25% of the workforce cut. The cause the company stated is a statement about the tail rather than the average — “We've determined the unpredictability in forecasting home prices far exceeds what we anticipated”.

The quantile row has a public specification. The M5 Uncertainty competition did not ask for a point forecast at all. It ran on Walmart sales from 2 March to 30 June 2020, over 42,840 hierarchical time series at a 28-day horizon, with $100,000 in prizes. It required nine quantiles per series — 0.005, 0.025, 0.165, 0.25, 0.5, 0.75, 0.835, 0.975, 0.995 — that is the median plus 50%, 67%, 95% and 99% central prediction intervals. One rule scored them, as a competing team's account records: “The precision of the probabilistic forecasts was evaluated using the Scaled Pinball Loss (SPL) function.” The organisers at the University of Nicosia say why the distributional contract was the point: “The majority of the 42,840 time series display intermittency (sporadic sales including zeros)”. The margins the loss had to resolve were narrow. The same team reports “Our overall result was a Weighted Scaled Pinball Loss of 0.162... ranked 6th out of 909 participants”, with the winner at 0.154 and the runner-up at 0.159.

Two losses may use different units, reductions, label spaces, and dataset sizes. A smaller numerical value does not automatically represent a better model across objectives. Within a single run, the loss and the decision quality can even move in opposite directions at the same time.

On CIFAR-100 with a 110-layer ResNet with stochastic depth, the test negative log-likelihood — the cross-entropy loss itself — overfits and rises during late training. Test error keeps falling, from 29% to 27%. The ICML 2017 paper that measured this says what the model is buying: “This phenomenon renders a concrete explanation of miscalibration: the network learns better classification accuracy at the expense of well-modeled probabilities.” Its abstract states the general finding: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated”. Researchers at Google's Brain Team retested the claim at NeurIPS 2021 and restated the same relationship, that “miscalibration worsen[s] even as classification error is reduced”. A reader watching only the loss curve would have concluded the model was getting worse. A reader watching only the error rate would have missed that its probabilities were.

Even within one loss family, a change in class weights or masking alters the scale. Compare behavior under a stable definition rather than memorizing a “good” loss value. The contract also covers what the model hands the loss. PyTorch's BCEWithLogitsLoss takes raw logits and applies the sigmoid itself. Its docstring says the class “combines a Sigmoid layer and the BCELoss in one single class”, and that this “is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, we take advantage of the log-sum-exp trick for numerical stability”. Passing probabilities to a logit loss runs cleanly and quietly trains something else.

The decision sequence below comes before any specialized loss. State what one output means — a logit per observation, as CheXpert's fourteen labels demand, rather than a softmax across them. Record the target uncertainty, including the positive, negative and uncertain scheme the labels actually carry. Decide which part of the error distribution matters: Zillow's average fit was survivable and its tail was not. Then test a simple baseline before anything exotic. In the Yahoo! challenge the organisers' pointwise GBDT baseline reached test ERR 0.46201, above the pairwise RankSVM baseline at 0.43680 and above the BM25F-SD baseline at 0.42853. Only then add specialized terms, and require evidence that each one fixes a measured failure.

A ranking system may train with a smooth pairwise surrogate and validate with top-k relevance. A detector may combine classification and localization losses but report matched precision and recall.

The Yahoo! Learning to Rank Challenge separated the two on purpose, and said so: “The NDCG scores were only provided for informational purposes: in order to determine the winners, the submissions were ranked according to their ERR score” (Chapelle and Chang, Yahoo! Labs, 2011). A surrogate is needed at all because the deciding measures are, in Burges's words, measures “which are not smooth functions of the model scores”. That is also why the winning Track 1 entry reached for lambda gradients, 8 of its 12 rankers being LambdaMART models trained directly on ERR or NDCG.

Keeping the two apart is healthy when the team states how they relate. It becomes dangerous when teams optimize a surrogate and never check the decision measure it was intended to improve — the failure Stahlberg and Byrne exposed by searching a likelihood objective to its exact optimum and finding the empty string there.