Large-Batch Training, Gradient Accumulation, and Scaling Limits

For a fixed number of examples, doubling batch size roughly halves optimizer updates. Each update uses a smoother gradient estimate, yet the model receives fewer opportunities to change.

Useful scaling occurs when parallel throughput outweighs the lost update frequency. Beyond a regime-dependent point, more examples per step add little new directional information.

Where that point sits has been measured, at a scale that leaves little room for argument. Six families of neural network. Three training algorithms. Seven data sets, with the learning rate, momentum and schedule retuned independently at every batch size. Shallue and colleagues published the result in the Journal of Machine Learning Research in 2019, and it rests on “a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads”. Two findings came out of it. The first is that the batch ceiling does not transfer: they report “extremely large variation between workloads”. The second is that the ceiling is not a quality cliff — they found “no evidence that larger batch sizes degrade out-of-sample performance”. The trade is real. The number at which it turns belongs to one system.

A system can win one and lose the other, and both halves have been measured on named runs rather than imagined.

Start with the hardware. ResNet-50 on ImageNet, a minibatch of 8192 spread across 256 GPUs, one hour of wall clock, roughly 90% scaling efficiency going from 8 to 256 GPUs. That run belongs to Goyal and eight Facebook co-authors, and it sits close to the top of what parallel work can pay.

On the statistical side the same axis moved the other way. Keskar and colleagues, at ICLR 2017, used 10% of the training set as the large batch against 256 points as the small batch under ADAM. Testing accuracy on their C4 network was 63.08% small batch against 57.81% large. On C1 it was 80.04% against 77.26%. Same knob, opposite sign of result, and neither run tells you what the other did.

The gradient sum is only one part of training state, and the gap between accumulation and a physical batch is not a theoretical worry. In October 2024 gradient accumulation was found to be mathematically non-equivalent to full-batch training in Hugging Face Transformers and other trainers. For token-level losses the cross-entropy mean was taken per micro-batch. Micro-batches holding different numbers of non-padding tokens were therefore weighted wrongly.

The clearest statement of the defect came from the library's own maintainers. “Gradient accumulation is supposed to be mathematically equivalent to full batch training; however, losses did not match between training runs where the setting was toggled on and off.” — Hugging Face, on its own blog, 16 October 2024.

An independent researcher, Benjamin Marie, reported it publicly, and Daniel Han of Unsloth replicated it. Marie measured loss gaps of 0.2–0.3 at 2048-token sequences, 0.1–0.2 at 512 tokens and 0.45–0.70 with highly diverse sequence lengths. After the fix the gap fell to about 0.0004. The fix was pull request #34191, opened on 16 October 2024 and merged on 17 October 2024: reduction="mean" became reduction="sum" divided by the true token count, so the denominator is the total loss over all micro-batches divided by the total non-padding tokens. A widely used trainer carried this for as long as nobody diffed the two runs.

A council doubles the size of its survey and halves how often it meets. Each survey is steadier, yet delayed decisions may slow adaptation.

The trade is the whole story. More evidence behind each decision, fewer decisions for a fixed amount of data, and a crossover point that has to be measured rather than assumed. The sections that follow are the measurements: where the crossover was found, how far it moved inside a single run, and what the two standard learning-rate rules were actually shown to do.

Both need warmup and empirical validation under the target optimizer. They are also not two rival theories. They are a theory and the heuristic that beat it, and both were written down in the same note.

The two rules were set side by side in April 2014, in a note called One weird trick for parallelizing convolutional neural networks. The theory is the square root: “Theory suggests that when multiplying the batch size by k, one should multiply the learning rate ε by √k to keep the variance in the gradient expectation constant”. What Alex Krizhevsky measured was something else. “Theory aside, for the batch sizes considered in this note, the heuristic that I found to work the best was to multiply the learning rate by k when multiplying the batch size by k. I can’t explain this discrepancy between theory and practice.” He held momentum at 0.9 and weight decay at 0.0005 across all batch sizes. He also marked the limit himself, in a footnote: the linear heuristic “does eventually break down for batch sizes larger than the ones considered in this note”.

The linear rule then acquired a measured envelope. Goyal and eight Facebook co-authors adopted “a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size” together with “a new warmup scheme that overcomes optimization challenges early in training”, and reported “no loss of accuracy when training with large minibatch sizes up to 8192 images”. ResNet-50 on ImageNet, minibatch 8192 across 256 GPUs, one hour, roughly 90% scaling efficiency from 8 to 256 GPUs, five-epoch gradual warmup. The warmup is there because the rule fails, and they say so: “As we discussed, for large minibatches (e.g., 8k) the linear scaling rule breaks down when the network is changing rapidly, which commonly occurs in early stages of training.” — Priya Goyal and eight Facebook co-authors, Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017.

The result was reproduced independently, in a section titled “Reproduce and Extend Facebook's result”: You and colleagues report batch-8192 peak test accuracy 0.727 against a batch-256 baseline of 0.730 at 90 epochs. The outer bound was fixed later by Shallue and colleagues, who record that Goyal et al. “increased the batch size from 64 to 8,192 with no loss in accuracy. However, their learning rate heuristic broke down for even larger batch sizes.” Inside 8192 the rule has evidence. Outside it, the paper claims nothing and neither should you.

The square-root rule has a derivation rather than an anecdote. At NIPS 2017, Hoffer and two Technion colleagues required the covariance of the update step to stay constant across mini-batch sizes and concluded that “we should increase the learning rate by the square root of the mini-batch size.” (η ∝ √M.) They also moved the blame for the large-batch generalization gap onto the number of weight updates rather than the batch size. With the square-root-scaled learning rate, Ghost Batch Normalization and regime adaptation applied cumulatively, their ResNet44/CIFAR-10 large batch rose from 86.10% to 93.07%, against a 92.83% small-batch baseline. Shallue and colleagues summarise the position from the outside: “Hoffer et al. (2017) argue that increasing the batch size need not degrade out-of-sample error at all, assuming training has gone on long enough.” The rule is still in production use. Ai2's OLMo batch-size warmup reports that “We also scale the learning rate by a factor of √2 when we increase the batch size, following the square-root scaling rule”.

Separate system speed from learning behavior. The shape you are looking for has already been characterized, which means the last step has a target instead of a hunch.

That shape was mapped with the learning rate, momentum and schedule retuned independently at every batch size — the tuning that makes a scaling curve mean anything. Shallue and colleagues describe what came out: “In all cases, as the batch size grows, there is an initial period of perfect scaling (b-fold benefit, indicated with a dashed line on the plots) where the steps needed to achieve the error goal halves for each doubling of the batch size. However, for all problems, this is followed by a region of diminishing returns that eventually leads to a regime of maximal data parallelism where additional parallelism provides no benefit whatsoever.”

So there are three regions, and the first one has a test you can run: does the step count to your error goal halve when the batch doubles? While it does, buy more parallelism. When it stops halving you are in the second region and paying partial rates. When the curve flattens you are in the third and paying for nothing. Where the boundaries fall is workload-specific. OpenAI puts it this way: “the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2”. Six model families, three training algorithms and seven data sets produced the same three-region shape and no shared number. The curve is the deliverable. The crossover is not.