Gradients, Batches, Epochs, and Learning Rate

The model predicts on a batch. The loss summarizes the error. Differentiation estimates how small parameter changes would affect that loss. The optimizer then combines this information with a learning rate, and possibly with memory from previous updates.

The parameters then move by a limited step. Repeating this procedure can reduce the training objective. No individual step guarantees a better validation result.

The optimizer in question is rarely anonymous in practice. Kingma and Ba published Adam in 2014, and the caption of Algorithm 1 hands the reader the numbers outright: “Good default settings for the tested machine learning problems are α = 0.001, β1 = 0.9, β2 = 0.999 and ϵ = 10−8.” The α is the learning rate. The two β values are the memory of previous updates. Mainstream tooling still ships exactly that: keras.optimizers.Adam defaults to learning_rate=0.001, beta_1=0.9, beta_2=0.999. A beginner's first script takes a step of 0.001 because a 2014 paper suggested it. Nothing about their problem was measured.

These terms describe different parts of the same process. The batch is the set of examples used to estimate one update. A step, or iteration, is one cycle that computes feedback and changes parameters. An epoch is enough steps to process approximately one full pass through the training set. The learning rate is a control on update magnitude, and it is often changed during training.

Batch size is not simply a quality dial. A small batch uses fewer examples per update. It needs less memory per step, produces a noisier gradient estimate, and gives more updates per epoch. A large batch averages feedback over more examples. It smooths the estimate, demands more memory, uses parallel hardware efficiently — and may need a learning-rate adjustment. A full batch has low sampling noise and few updates per epoch, and is expensive enough to be rare in modern large-scale training.

That last caution on the large-batch side is not a vague warning. It has a name, a rule and a measured cost. In 2017 Facebook trained ResNet-50 on ImageNet with a minibatch of 8,192 on 256 GPUs, in one hour. Goyal and eight colleagues state the whole method in one line: “Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.” Moving from 64 to 8,192 examples per update means multiplying the learning rate by the same factor, with a gradual warmup over the first 5 epochs to survive the start.

The result: 23.74% ±0.09 top-1 validation error against a 23.60% ±0.12 small-minibatch baseline. A batch multiplied by 32, a learning rate multiplied by 32, and an accuracy gap inside the run-to-run error bars. Shallue and colleagues at Google Brain later restated that 64→8,192 result along with its limit: the same heuristic broke down for even larger batch sizes. The rule is real and it is bounded. That is a different sentence from “may need learning-rate adjustment”.

Optimization problems often reveal themselves before final evaluation. Two of the patterns below are not lore passed between practitioners. They are dated, published schedules with step counts attached.

High training error can indicate insufficient capacity, weak features, noisy targets, a poorly aligned loss, or optimization that never found a good fit. Increasing model size without separating these causes can waste time.

A tiny-sample fit and a learning-rate sweep help distinguish implementation and optimization problems from representational limits. So do gradient checks and a simpler baseline.

The cheapest version of that sweep was published in 2015. Leslie Smith, at the US Naval Research Laboratory, called it the “LR range test”. It runs the model for a few epochs. During the run the learning rate rises linearly from a low value to a high one. Then it plots accuracy against rate. Read off the rate at which accuracy “starts to increase”, and the rate at which it “slows, becomes ragged, or starts to fall”. Those two numbers bracket the usable range. On CIFAR-10 with Caffe's stock architecture that gave him 0.001 and 0.006. One short run separates a model that looks incapable of learning from a model whose step size was simply wrong. It does that before anyone buys more parameters.

A probe should produce a diagnosis. It should not produce a pile of unrelated runs. Fix the evidence — one dataset version, split, model and metric. Choose a rate range spanning several orders of magnitude rather than tiny cosmetic differences. Watch the early curves, recording loss, gradient or update norms, and numerical warnings. Compare equal budgets, meaning the same steps or examples processed. Then confirm generalization with validation evidence rather than training speed alone.

The last two are the steps people skip, and there is a large piece of evidence about what skipping them costs. One Google Brain study measured 71,638,836 loss values across 168,160 trained models and 454 workload/batch-size combinations. Shallue and five co-authors published it in 2019, and they say what all that measurement was for: “Specifically, we show that assumptions about computational budgets and the procedures for selecting metaparameters at different batch sizes can explain many of the disagreements in the literature.”

Underneath the disagreements they found one universal shape. Raising the batch size first cuts the required steps proportionally, then hits diminishing returns, then buys nothing at all. The turning point is set by the workload rather than by any fixed number. OpenAI researchers arrived at the same boundary from another direction in 2018, predicting the largest useful batch size from a measurable gradient noise scale. Holding the budget and the tuning procedure fixed is therefore not fastidiousness. It is the difference between producing a result and producing a disagreement.

A perfectly optimized objective on biased, leaked, or outdated data can produce a poor system. Conversely, modest optimization may be sufficient. That is the case when the signal is strong and the baseline is weak.

Later training paths will cover optimizer families and schedules in detail. At this stage, remember that optimization controls how a model fits the specified problem. It does not validate the problem itself.

The gap between the two was made concrete at ICLR 2017. Keskar, Nocedal and three co-authors moved from the customary 32 to 512 examples per update to a much larger batch. They observed “a degradation in the quality of the model, as measured by its ability to generalize”. They presented evidence that large-batch training converges on sharp minimisers of the training and testing functions, while small-batch training settles into flat ones. The training objective was not the thing that failed. Both regimes optimised it. Only one of them produced a model worth shipping.