Skip to content
AI.info

Training and optimization

Mini-Batches, Gradient Variance, and Effective Batch Size

Learn how batch size, composition, accumulation, correlation, and normalization determine the statistical and numerical character of an update.

By the end you can

Visual

One “batch size” can hide three different quantities

Separate these quantities before applying a scaling rule. In a production system each level answers to a different constraint, and each one is tuned on its own.

Megatron-LM reports all of them in a single configuration. Twelve authors at NVIDIA, Stanford and Microsoft Research published it in 2021. The micro-batch is chosen for throughput: “In our experiments, the optimal value of the microbatch size is problem-dependent and can increase throughput by 15%.” For their 91-billion-parameter configuration the best micro-batch size was 2.

The global batch answers to something else. It bounds how wide the data-parallel dimension can be at all: “GPT-3 was trained to convergence with a batch size of 1536. Data parallelism thus supports parallelization to only 1536 GPUs; however, roughly 10,000 GPUs were used to train this model in a reasonable amount of time”. Two levels, two constraints, one word. The composed configuration reached 502 petaFLOP/s on 3072 A100 GPUs, 52% of theoretical peak.

FigureHierarchy · 4 levels

  • Micro-batch per device

    Examples processed in one forward and backward pass on one worker.

    • Accumulated batch per worker

      Micro-batches whose gradients are combined before one optimizer step.

      • Global nominal batch

        Accumulated examples across all synchronized workers.

        • Effective independent evidence

          The diversity remaining after correlation, duplication, padding, and sampling weights.

Bigger batches change both statistics and scheduling

A larger batch usually lowers sampling variance and keeps the hardware busier, until memory runs out or the workers spend more time talking to each other than computing. It also produces fewer optimizer steps for the same number of examples.

Those effects are entangled. Change batch size while keeping epochs fixed and the step count moves, and so do schedule timing, regularization, and often the amount of augmentation randomness. Goyal and colleagues wrote the compensation down in 2017, in one line: “When the minibatch size is multiplied by k, multiply the learning rate by k.” With that rule and a warmup phase they report “no loss of accuracy when training with large minibatch sizes up to 8192 images”. The run trained “ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour”, at “∼90% scaling efficiency when moving from 8 to 256 GPUs”.

Nine authors, one architecture, one dataset. The point is not the record but the bookkeeping it required. The batch moved, and the learning rate and the warmup had to move with it before the comparison meant anything.

Batch size is not a memory setting; it changes the experiment.

Analogy

Polling a population with clustered respondents

Thousands of answers drawn from a few households make a large response count. Shared circumstances reduce the independent information behind that count. Survey statistics has been pricing that reduction for decades, with a name and a formula.

The name is the design effect, and the World Health Organization works the arithmetic in its cluster-survey reference manual: “Assume we will collect data from an average of m=7 respondents per cluster and assume an intracluster correlation coefficient of 1/3, so the design effect will be 3.” The worked example starts from an effective sample size of 103 — the count a simple random sample would need — and multiplies it by that design effect. So 309 interviews have to be collected to buy the precision of 103 independent ones. The manual says plainly what the multiplier is for, “a multiplier required because this is a cluster survey and vaccination status is likely to be spatially correlated”, and notes that earlier guidelines assumed a design effect of 2.

The same two definitions run US health statistics. The design effect is “the ratio of the variance of a statistic which accounts for the complex sample design to the variance of the same statistic based on a hypothetical simple random sample of the same size”, and the effective sample size is “an actual sample size divided by the design effect”. Those are the National Center for Health Statistics' words for NHANES, where design effects are typically greater than 1. A batch of correlated examples is a cluster sample that nobody has computed a design effect for.

Beyond a task-dependent region, increasing batch size can reduce gradient variance without reducing the number of updates needed proportionally. Hardware speedup then stops translating into equal training speedup, and there is no universal critical batch size to look up. OpenAI named the statistic in 2018. It shows that “a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training”. It also reports that “the noise scale increases as the loss decreases over a training run”. The quantity you would like to set once moves while you train.

Count evidence by diversity and validity, not only rows or tokens.

Case

Seventy-one million measurements, and no critical batch size to borrow

Somebody ran the controlled version of the batch-size experiment, and released the raw material with it. Shallue and colleagues published, in the Journal of Machine Learning Research in 2019, “a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads”.

Two results survive that volume. The first is that there is nothing to borrow. Across the workloads they “find extremely large variation between workloads”, so no single critical batch size holds, and a scaling curve measured on someone else's model is not a constant you can carry over. The second is negative, and stated as such: they “find no evidence that larger batch sizes degrade out-of-sample performance”. Where earlier studies had disagreed, the disagreement tracked differences in tuning and in compute budget.

Example

Nominal examples are not always independent evidence

These examples can make a global batch look larger than it behaves. The survey formula gives the discount a shape: divide the nominal count by a design effect that grows with cluster size and with within-cluster correlation.

  • Video frames: adjacent frames share nearly identical content, so 256 frames may contain far fewer independent events — the shot is the cluster, and WHO's worked case shows the price, 7 members per cluster at an intracluster correlation of 1/3 giving a design effect of 3.
  • Patient records: several visits from one patient create correlated gradients unless grouping is handled deliberately; the patient is the cluster exactly as the household is in a coverage survey, and what counts is, in the NCHS definition, “an actual sample size divided by the design effect”.
  • Oversampling: repeating the same rare examples raises the nominal count without adding new information. It moves the numerator of that ratio while the evidence behind it stays where it was.
  • Language sequences: many padded tokens contribute no target signal, shrinking the valid-token batch — the same quantity Hugging Face's corrected loss divides by, the total number of non-padding tokens across the accumulated step rather than a mean of per-micro-batch means.
  • Distributed shards: workers reading similar or overlapping shards reduce the diversity expected from all-reduce, the way clustered fieldwork leaves NHANES design effects typically greater than 1.

Key idea

Gradient accumulation is not universally identical to one physical batch

Equivalence requires the same reduction, examples, randomness, and parameter state across micro-batches. BatchNorm statistics, dropout masks, sequence padding, gradient clipping, and optimizer schedules can break it. This is not a hypothetical caution. It is a dated incident with a patch.

On 16 October 2024 Hugging Face published a fix for its own trainer. “Gradient accumulation is supposed to be mathematically equivalent to full batch training; however, losses did not match between training runs where the setting was toggled on and off.” The default loss in transformers averaged per-micro-batch mean losses. Here is the maintainers' own statement of the correct computation: “the correct loss should be computed by the total loss across all batches in a gradient accumulation step divided by the total number of all non padding tokens in those batches. This is not the same as the average of the per-batch loss values.” The defect was reported by Unsloth and @bnjmn_marie and patched with a reduction="sum"/num_items change in under 24 hours.

The defect class outlived that fix. Limozin and colleagues found the same mean-of-per-mini-batch-means bug still live in OpenRLHF and Llama-Factory, with verl fixing it in November 2025. Alongside it sat a DeepSpeed CPU-offload bug, introduced in September 2024 in DeepSpeed PR #6550, that let only the first micro-batch's gradients reach the optimizer. Together the two deflated a published SFT baseline on Qwen2.5-Math-7B. OpenRLHF averaged 48.3. The loss fix raised it to 49.1, the optimizer fix to 53.4, and both together to 54.0, against an independently implemented verl baseline at 53.8. Repaired, the plain SFT-then-RL pipeline beats the best published mixed-policy method by +3.8 points on in-distribution math benchmarks with Qwen2.5-Math-7B, and by +22.2 points with Llama-3.1-8B. Two years of published comparisons rested on a reduction.

Accumulation also changes when metrics, EMA updates, or learning-rate steps occur. The stateful layer the algebra most often forgets is normalization. Hoffer and colleagues note that BN “is bounded to depend on the choosen batch size”, which is why they had to invent a batch-independent variant before large-batch runs would compare fairly. Test the code you are actually running instead of assuming the algebra carries over.

Matching the summed gradient is necessary, but stateful training behavior can still differ.

Comparison

Small and large batches create different operating regimes

The useful choice depends on model, data, hardware, and schedule. The risk listed against the smaller batch — unstable normalization statistics — is not a qualitative worry. Wu and He measured the curve it follows.

In 2018 they swept ResNet-50 on ImageNet across per-GPU batch sizes of 32, 16, 8, 4 and 2. Validation error ran 23.6%, 23.7%, 24.8%, 27.3% and 34.7%. Batch-independent Group Normalization over the same sweep held at 24.1%, 24.2%, 24.0%, 24.2% and 24.1%. Their abstract states the gap: “On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants.”

One detail decides which of this lesson's quantities the layer is actually reacting to. In those runs the BN mean and variance were computed within each GPU and not synchronized across the 8 GPUs used. The normalization statistics saw the micro-batch, not the global batch. An accumulated batch of 8192 built from micro-batches of 2 is, for this layer, a batch of 2.

FigureComparison · 3 columns

Smaller batch

More updates per example budget and higher sampling variability.

  • Benefit: frequent feedback
  • Benefit: lower activation memory
  • Cost: weaker hardware utilization
  • Risk: unstable normalization statistics

Larger batch

Fewer, smoother updates with more parallel work.

  • Benefit: throughput at scale
  • Benefit: steadier gradient estimates
  • Cost: communication and memory
  • Risk: diminishing optimization returns

Accumulated batch

Several micro-batches feed one optimizer update.

  • Benefit: fit large nominal batches
  • Benefit: control synchronization frequency
  • Cost: longer delay between updates
  • Risk: not identical under stateful layers

Steps

Run a controlled batch-size study

Change one regime carefully and preserve comparable budgets.

Step 1 is not pedantry, because one model's batch size is on the public record in two different units. OpenAI reports GPT-3's in tokens: 0.5M tokens for the 125M–760M models, then 1M, 2M and 3.2M tokens for the 175B model, with a 2048-token context. The table caption says what the numbers are: “Sizes, architectures, and learning hyper-parameters (batch size in tokens and learning rate) of the models which we trained. All models were trained for a total of 300 billion tokens.” NVIDIA's Megatron-LM paper reports the same model's batch as 1536, meaning 1536 sequences. Neither is wrong. A comparison that mixes them is off by three orders of magnitude.

Steps 2 and 5 are the Megatron arithmetic and its consequences: micro-batch × accumulation steps × synchronized workers, with the global batch capping the width of data parallelism. GPT-3's 1536 supports 1536 workers, and roughly 10,000 GPUs were actually used. The systems number a well-composed configuration bought was 502 petaFLOP/s on 3072 A100 GPUs, 52% of theoretical peak.

Step 4 is where the statistic from the analogy section stops being theory. OpenAI spends one sentence on it in the GPT-3 paper: “We measure the gradient noise scale during training and use it to guide our choice of batch size”. The measurement was made during the run, on the workload. It changes during the run, and it belongs to the workload.

FigureProcess · 6 steps

  1. 1. Define the batch unit

    State whether counts refer to samples, valid tokens, pixels, pairs, or sequences.

  2. 2. Record the global formula

    Multiply micro-batch, accumulation steps, and synchronized workers.

  3. 3. Match processed evidence

    Compare runs at equal examples or tokens, not merely equal epochs.

  4. 4. Retune step-linked settings

    Revisit learning rate, warmup, decay, clipping, and EMA cadence.

  5. 5. Measure systems behavior

    Track throughput, memory, communication, and time to validation target.

  6. 6. Inspect slice effects

    Check whether rare groups disappear from larger or more homogeneous batches.

Position

The 5% generalization gap is real, and it is a budget artefact

Keep the batch small, because big batches generalize worse. The advice arrives early and gets repeated as a property of large-batch training, the way a constant gets repeated. It is not invented. It has a source, and the source has a number. In 2017 Keskar and colleagues reported it: “In our experiments, we have found the drop in generalization (also called generalization gap) to be as high as 5% even for smaller networks.” They trained six networks with ADAM, using 256 examples per small batch against 10% of the training set per large batch. For network C1 on CIFAR-10 the testing accuracy was 80.04% ± 0.12% small-batch against 77.26% ± 0.42% large-batch; for C4, 63.08% ± 0.5% against 57.81% ± 0.17%. It replicates, too. Hoffer and colleagues re-ran Keskar's own F1 and C1 networks and saw the same thing, 98.27% against 97.05% on F1/MNIST and 87.80% against 83.95% on C1/CIFAR-10.

What got dropped in transmission is the protocol. Those runs were made “without any budget or limits” — the same epochs for both arms, and therefore far fewer updates in the large-batch arm than in its small-batch twin. Change batch size while holding epochs fixed and several things move at once: step count, schedule timing, regularization, and often augmentation randomness, as this lesson lists. A large-batch run left otherwise untouched is a differently trained model, not the same model seeing more examples per step. When a validation gap appears, batch size is the variable that got named. It is also the one that was least alone.

The other variables can be moved back, one at a time, and Hoffer and colleagues published the column of numbers. On ResNet44/CIFAR-10, validation accuracy fell from 92.83% with the small batch to 86.10% with the large one — the folklore, reproduced. Adjusting the learning rate returned 89.30%. Ghost Batch Normalization, which exists precisely because BN “is bounded to depend on the choosen batch size”, returned 90.50%. Adapting the number of updates returned 93.07%, above the small-batch baseline the gap was measured against. Across their networks the gap shrank from about 5% to 1–2%. Their own summary is flat: “There is no inherent "generalization gap": large-batch training can generalize as well as small batch training by adapting the number of iterations.” The abstract puts the diagnosis in one clause — the gap “stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting the training regime used”.

The same repair works at the other end of the scale. Goyal and colleagues multiplied the learning rate by k, added warmup, and report “no loss of accuracy when training with large minibatch sizes up to 8192 images”. Shallue and colleagues then went looking for the penalty across 71,638,836 loss measurements over 168,160 models and 35 workloads, and did not find it: they “find no evidence that larger batch sizes degrade out-of-sample performance”. The claim that survived twenty times the evidence is the one about the training regime, not the one about the batch.

A real limit exists, and it is not generalization. Beyond a task-dependent region, more examples per batch stop reducing the updates needed in proportion, and hardware speedup stops becoming training speedup. OpenAI's gradient noise scale predicts the largest useful batch size in many domains, and it rises as the loss falls during a run. Shallue's answer to where the limit sits is that they “find extremely large variation between workloads”. So the limit is real, it moves inside a single run, and it belongs to the workload you have.

Key takeaways