Training and optimization
Gradient Clipping, Exploding Updates, and Trustworthy Norms
Compare global-norm, value, adaptive, and per-parameter clipping while preserving diagnostic evidence about spikes and invalid gradients.
By the end you can
- Distinguish exploding gradients from large but legitimate gradients
- Compare global-norm clipping with value and group-specific strategies
- Explain the correct order of unscaling, clipping, and optimizer updates
- Use clipping telemetry to diagnose rather than normalize instability
Example
One batch can move a model farther than a thousand ordinary batches
PaLM 540B was trained with the guardrail on the whole way through: “global norm gradient clipping … with a value of 1.0 for all models”. It spiked anyway. Chowdhery and colleagues record what that guardrail bought them: “For the largest model, we observed spikes in the loss roughly 20 times during training, despite the fact that gradient clipping was enabled.” The smaller models in the same family showed none.
The mitigation was operational, not analytic. Restart from a checkpoint about 100 steps before the spike, then skip the next 200–500 data batches. Then came the experiment that matters. They re-ran those same batches from an earlier checkpoint, and the model trained through them cleanly. Hence the conclusion that “spikes only occur due to the combination of specific data batches with a particular model parameter state”. Not poisoned data. Not a broken pipeline. Not something a threshold of 1.0 was able to absorb. A batch met a parameter state, and neither one alone was at fault. Keep the whole list of candidate causes open when a norm jumps.
- Long sequence: Recurrent or attention paths accumulate unusually large sensitivity for one example.
- Outlier target: A regression residual dominates a mean reduction and creates a rare update.
- Loss-scale change: A reduction or class weight increases every gradient without changing the optimizer rate.
- Numerical overflow: Mixed-precision gradients become infinite before the clipping operation sees them.
- Real boundary case: A rare but valid example genuinely demands a substantial model correction — or, as in PaLM 540B, an otherwise ordinary batch meets a particular parameter state and neither one alone is at fault.
Clipping changes the update, not the underlying objective
Global-norm clipping rescales the full gradient vector when its norm exceeds a threshold. Value clipping truncates individual coordinates.
Both interventions alter optimization. They can prevent catastrophic steps, yet frequent activation may bias learning and conceal a defective loss or data pipeline. The PaLM run is the reminder that the reverse also holds. A guardrail can be switched on, correctly configured, and still leave twenty spikes in the loss curve for someone to explain.
Pascanu and colleagues proposed the norm rule in 2013. They describe it as “a gradient norm clipping strategy to deal with exploding gradients”. GPT-3 used the same guardrail at scale seven years later. Its published recipe clips “the global norm of the gradient at 1.0”.
Clipping is a guardrail whose activation rate belongs on the dashboard.
Visual
Safe ordering under mixed precision
The threshold should apply to gradients in their intended scale. Micikevicius and colleagues, who made loss scaling standard practice, say exactly where in the step to restore it: “It is simplest to perform this unscaling right after the backward pass but before gradient clipping or any other gradient-related computations, ensuring that no hyper-parameters (such as gradient clipping threshold, weight decay, etc.) have to be adjusted.” That is Mixed Precision Training, from 2018. Its authors trained networks with constant loss-scale factors ranging from 8 to 32K, against an FP16 maximum of 65,504, and skipped the weight update entirely whenever an overflow was detected.
Framework documentation states the consequence of getting the order wrong. PyTorch 2.14's mixed-precision guide, under its Gradient clipping heading: “If you attempted to clip without unscaling, the gradients' norm/maximum magnitude would also be scaled, so your requested threshold (which was meant to be the threshold for unscaled gradients) would be invalid.” The prescribed sequence is the one below — “Calling scaler.unscale_(optimizer) before clipping enables you to clip unscaled gradients as usual”. A threshold applied to scaled gradients does not mean 1.0. It means whatever the loss scale happened to be on that step.
- 1
Scale the loss if required
FP16 training may multiply the loss before backpropagation.
- 2
Backpropagate scaled gradients
The graph produces values enlarged by the current loss scale.
- 3
Unscale and check finiteness
Restore true gradient scale and detect overflow.
- 4
Apply the clipping rule
Measure and limit gradients using the declared norm or value policy.
- 5
Run the optimizer step
Update parameters only when the gradients are valid.
A circuit breaker that prevents one destructive surge
When a dangerous current spike arrives, the breaker opens before the equipment burns. It protects the system but does not explain why the surge occurred.
A breaker cuts the circuit. Clipping lets the step through at a smaller size, so every activation leaves a rescaled update in the model rather than no update at all. Protection, telemetry, and root-cause analysis are all still the job. PaLM 540B shows that the analysis is not optional. The guardrail there was enabled throughout, and the spikes still had to be traced to a batch-and-state combination and stepped around by hand.
A guardrail is valuable only when its activations remain visible.
Comparison
Different clipping rules preserve different information
Choose the rule according to the failure and the parameter structure. Global-norm clipping rescales everything together and preserves direction. Value clipping bounds each coordinate and can rotate the vector. Two published alternatives show what a different rule buys and what it costs.
One fixed global number would not hold, and that is why Adaptive Gradient Clipping exists. DeepMind introduced it in 2021, and said what forced the move: “Empirically, we found that while this clipping algorithm enabled us to train at higher batch sizes than before, training stability was extremely sensitive to the choice of the clipping threshold, requiring fine-grained tuning when varying the model depth, the batch size, or the learning rate.” AGC therefore clips unit-wise, on the ratio of gradient norm to parameter norm, with ε = 10⁻³. The threshold λ was swept over [0.01, 0.02, 0.04, 0.08, 0.16] at batch sizes from 256 to 4096, and “smaller (stronger) clipping thresholds are necessary for stability at higher batch sizes”. The resulting NFNets reached 86.5% ImageNet top-1.
A fourth documented rule clips before averaging rather than after, and it is not there for stability at all. Abadi and colleagues, in Deep Learning with Differential Privacy in 2016, bound every per-example gradient by g/max(1, ‖g‖₂/C) before it is averaged. They mark the difference from ordinary practice themselves: “We remark that gradient clipping of this form is a popular ingredient of SGD for deep networks for non-privacy reasons, though in that setting it usually suffices to clip after averaging.” They pick C from the median of the unclipped gradient norms, and use a reference bound of C = 4 on MNIST. They also state the price without hedging: “clipping destroys the unbiasedness of the gradient estimate”.
A standards body describes the same mechanism independently. NIST Special Publication 800-226, on evaluating differential privacy guarantees, published in 2025: “To enforce an upper bound on sensitivity, the algorithm clips the gradient associated with each training example”. The same document names the bias a clipping parameter introduces — “reducing large values while preserving small ones”. That is the honest one-line summary of every rule in this comparison, stability rules included.
Global norm
Scale all gradients together to satisfy one norm threshold.
- Preserves: overall direction
- Changes: magnitude
- Use: common deep training guardrail
- Risk: one tensor shrinks every module
Value clipping
Clamp each coordinate to a numeric interval.
- Preserves: bounded coordinates
- Changes: vector direction
- Use: selected recurrent recipes
- Risk: widespread distortion
Group or adaptive clipping
Set thresholds relative to modules or parameter norms.
- Preserves: local scale awareness
- Changes: cross-group balance
- Use: heterogeneous models
- Risk: more policy complexity
Key idea
A threshold copied from another model has little meaning by itself
Gradient norm depends on loss reduction, parameter count, model scale, precision, and distributed aggregation. The same threshold can clip every step in one system and never activate in another. The authors who introduced the rule said so first. Pascanu and colleagues, in 2013: “One good heuristic for setting this threshold is to look at statistics on the average norm over a sufficiently large number of updates.” The ICML version adds a band around that average — “values from half to ten times this average can still yield convergence, though convergence speed can be affected”. Their own toy experiments used a threshold of 1.0. The instruction was to measure, then choose.
What travels instead is the number. GPT-3 (175B) clipped every model size the same way: “we clip the global norm of the gradient at 1.0”. DeepMind's Gopher paper did not. “We clip gradients based on the global gradient norm using a clipping value of 1. However, for the 7.1B model and for Gopher we reduce this to 0.25 for improved stability.” The same nominal guardrail, a factor of four apart between two frontier runs of the same era, and changed by scale inside a single codebase. Estimate a healthy distribution first. Then choose a threshold with a stated purpose, and revisit it after objective or batch changes.
Clipping before gradient synchronization can differ from clipping the aggregated gradient. Under sharding it is not even the same function. PyTorch documents that FullyShardedDataParallel.clip_grad_norm_ replaces the ordinary call: “If at least some FSDP instance uses a sharded strategy (i.e. one other than NO_SHARD), then you should use this method instead of torch.nn.utils.clip_grad_norm_() since this method handles the fact that gradients are sharded across ranks.” The same entry warns that “This needs to be called on all ranks since it uses collective communications.” Hugging Face Accelerate says its Accelerator.clip_grad_norm_ “Should be used in place of torch.nn.utils.clip_grad_norm_”, and its documented example guards the call with “if accelerator.sync_gradients”. That is the accumulation boundary, not every micro-batch. Choose one semantic policy, implement it deliberately, and test it against a small single-worker reference. PyTorch measures that norm “as if the norms of the individual gradients were concatenated into a single vector”, with the default power 2.0. Keras 3 splits the two semantics into separate arguments: clipnorm per weight, global_clipnorm across all of them.
A threshold is a measurement of your own run: 1.0 for GPT-3, 0.25 for Gopher, 0.3 for OPT-175B.
Steps
Investigate a clipping-heavy run
Frequent clipping is a symptom worth localizing, and one 175B run published the entire trail. Meta's OPT paper reports that “Early in training, we also noticed that lowering gradient clipping from 1.0 to 0.3 helped with stability; see our released logbook for exact details”.
The logbook is the evidence, and it is a primary artefact rather than a summary. 114 pages of engineering log, released with the model, with entries running from 2021-10-20 to 2022-01-13. The decision has a timestamp and a run number — entry 2021-11-23 10:50am [Myle]: Run 12.16 — and one line: “Decided to reduce clipping to 0.3 and relaunch from 24.5K checkpoint”. The resulting checkpoint path carries the tag cl0.3, so the threshold that produced the weights is recoverable from the filename. The same run took at least 35 manual restarts and cycled over 100 hosts in two months.
That is what a clipping-heavy investigation actually looks like: change the threshold, relaunch from a known-good checkpoint, and record which run number it was so the next person can tell the two apart. The steps below are that loop.
1. Log pre-clip norms
Preserve the true distribution and identify the batches that trigger clipping.
2. Break down by module
Find whether one tensor or branch dominates the global norm.
3. Inspect triggering examples
Check labels, lengths, masks, weights, and outlier targets.
4. Test scale interventions
Compare lower rate, robust loss, reduction fixes, or normalization changes.
5. Reassess the guardrail
Keep clipping only if it protects a valid regime without masking chronic failure.
Key takeaways
- Gradient clipping is an explicit modification of updates that can prevent catastrophic steps but cannot repair the objective: PaLM 540B ran global-norm clipping at 1.0 and still took roughly 20 loss spikes, none of which appeared in the smaller models.
- Global-norm clipping preserves vector direction, value clipping bounds coordinates and can rotate it, AGC clips unit-wise on the ratio of gradient norm to parameter norm with ε = 10⁻³, and DP-SGD clips per example before averaging — which, as Abadi and colleagues state, destroys the unbiasedness of the gradient estimate.
- Mixed-precision gradients should be unscaled and checked before a threshold is applied: the Mixed Precision Training paper places the unscaling right after the backward pass, and PyTorch warns that a threshold applied to scaled gradients is invalid.
- Pre-clip norms, triggering examples, module contributions, and activation frequency turn clipping into diagnostic evidence — OPT-175B's released 114-page logbook shows what that record looks like, down to Run 12.16 on 23 November 2021.
- Thresholds are measured, not copied: Pascanu and colleagues prescribe statistics on the average norm over many updates, with half to ten times that average still converging, while GPT-3 used 1.0, Gopher 0.25, and OPT-175B moved from 1.0 to 0.3 mid-run.
- Distributed and accumulated training must define whether clipping applies locally, globally, per micro-batch, or after accumulation — FSDP requires its own clip_grad_norm_ on all ranks because the global norm needs collective communication, and Accelerate's example clips only when sync_gradients is true.