Loss Functions as Learning Contracts

Training needs a scalar objective whose local changes can be differentiated. The loss compares model outputs with targets or other training signals, and then assigns a penalty.

A lower loss means the model better satisfies that mathematical contract. The contract covers the measured data. It does not automatically mean the product became more useful, fair, or safe.

Satisfying the contract well does not make the outputs mean what they look like they mean. A 2017 ICML paper on calibration put it flatly: “modern neural networks, unlike those from a decade ago, are poorly calibrated”. Its Figure 1 sets the two failures side by side on CIFAR-100. A 5-layer LeNet makes 44.9% error and is nearly calibrated. A 110-layer ResNet makes 30.6% error and is badly over-confident. The more accurate model is the one whose confidence is worth less. The same paper names what drives it: “depth, width, weight decay, and Batch Normalization are important factors influencing calibration”. Cross-entropy rewards putting probability mass on the correct class. It does not promise that an output of 0.9 is right nine times in ten.

The remedy was a separate step, fitted after training: “on most datasets, temperature scaling — a single-parameter variant of Platt Scaling — is surprisingly effective at calibrating predictions.” That repair has a domain of validity, and the domain ends where the training distribution does. A NeurIPS 2019 benchmark measured temperature scaling under corrupted data. Its expected calibration error rises sharply as corruption intensity increases, and under shift nearly every other method beats it on Brier score. The benchmark states the limit in one sentence: “An important observation is that while calibrating on the validation set leads to well-calibrated predictions on the test set, it does not guarantee calibration on shifted data.” A fix fitted on the validation set is another contract. It too covers only the data it was measured on.

Most training objectives pass through several distinct decisions. Each one is a place where the number the optimizer sees can stop describing the outcome anyone wanted: the head's output format, the target encoding, the per-element scoring rule, the treatment of masked or reweighted entries, and the aggregation that collapses everything into a single scalar.

Similar-looking predictions can imply different penalties and therefore different parameter updates.

The robust middle option is not a modern patch on squared error. Peter J. Huber published it in 1964, in The Annals of Mathematical Statistics. scikit-learn implements the idea as H_epsilon(z) = z² for |z| < epsilon and 2·epsilon·|z| − epsilon² otherwise. Epsilon is the exact point where the penalty stops squaring and starts growing linearly — the point past which one outlier can no longer buy unlimited influence over the fit.

That is why the phrase “different derivative near zero or threshold” is worth a number rather than a warning. The scikit-learn user guide gives the setting practitioners should use: “It is advised to set the parameter epsilon to 1.35 to achieve 95% statistical efficiency.” The trade is legible in one line. On clean Gaussian data the Huber fit retains 95% of the efficiency of least squares, and in exchange the tail of the residual distribution stops dictating the answer. Squared error, absolute and Huber losses, and cross-entropy are not three styles of the same statement. They estimate different target statistics, and they hand the optimizer different gradients for the same visible error.

Summing per-example losses makes gradient magnitude grow with the batch or sequence length. Averaging stabilizes that scale across batch sizes, but it leaves masking and variable-length data to complicate the denominator.

A token-level mean weights long documents more heavily than a document-level mean. State the unit being averaged before comparing experiments.

This is not a classroom caveat, and it is recent. In October 2024 the mean-reduced cross-entropy default in Hugging Face Transformers meant that gradient accumulation was not mathematically equivalent to full-batch training for causal LM. Averaging per-micro-batch means whose non-padding token counts differ is not the same as one mean over all the tokens. Hugging Face published the diagnosis and the repair on 16 October 2024: “To be precise, for gradient accumulation across token-level tasks like causal LM training, the correct loss should be computed by the total loss across all batches in a gradient accumulation step divided by the total number of all non padding tokens in those batches. This is not the same as the average of the per-batch loss values.” Sum the token losses; divide by the total number of non-padding tokens. PyTorch's torchtune landed the identical normalization in PR #1875, “Normalize CE loss by total number of (non-padding) tokens”, merged 25 October 2024. Two of the most-used training stacks shipped the wrong denominator. The mistake was in the reduction, not in the model or the optimizer.

Reweighting can address priorities, but it also changes the objective being estimated. The frameworks state the mechanism in their own reference pages rather than leaving it to intuition. PyTorch's BCEWithLogitsLoss works the arithmetic. For a dataset with 100 positive and 300 negative examples, set pos_weight to 300/100 = 3, and the loss then “would act as if the dataset contains 3×100 = 300 positive examples”. Nothing on disk changed. The population the gradient sees did. PyTorch also states the direction of the lever: pos_weight above 1 increases recall, below 1 increases precision. TensorFlow documents the same lever as an explicit trade: “This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.”

Many training failures are objective failures in disguise.

A change to the objective can pay in one currency and charge in another. Label smoothing has both sides of its ledger measured. It arrived at CVPR 2016 with epsilon = 0.1 over the K = 1000 ImageNet classes, training against a weighted average of the hard targets and the uniform distribution over labels. What that bought is stated plainly: “For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error (cf. Table 3).” Table 3 carries the numbers behind the sentence — top-1 error of 23.1% without smoothing against 22.8% with it, and top-5 error of 6.1%.

A NeurIPS 2019 paper by Hinton and two colleagues returned to the method and confirmed a second benefit, since “in addition to improving generalization, label smoothing improves model calibration”. Then they measured the charge. They found that “if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective”. Smoothing “encourages the representations of training examples from the same class to group in tight clusters”, and so loses “information in the logits about resemblances between instances of different classes”. Two tenths of a point of top-1 error bought, and a model that has become a worse teacher. The loss curve reports the purchase. It has no column for the bill.

So audit the objective before the optimizer. Name the behavior you want and the unit it applies to. Inspect the targets for delay, missingness and ambiguity. Check that the output format matches what the loss expects. Recompute the reduction across examples, classes and tokens. Then confirm on held-out data that the lower loss corresponds to better decisions.