Towards Full Pipeline FP8 Reinforcement Learning for LLMs

AI summary

Overview

Research area: Efficient training systems for large language models — specifically low-precision (FP8) reinforcement learning pipelines used for post-training LLMs.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (policy gradients, importance ratios, trust regions, advantage estimation), FP8 quantization schemes, and the train-inference mismatch problem in RL pipelines.

Scope: The paper diagnoses a training-instability failure specific to full-pipeline FP8 reinforcement learning and proposes a clipping correction — Calibrated Clipping — to restore stability and BF16-comparable performance.

What This Paper Is About

FP8 quantization can speed up RL training for LLMs, but keeping the whole pipeline stable under FP8 is still hard. Earlier work attributed FP8 RL problems mainly to mismatches between the training and inference engines, addressed with corrections such as TIS. This paper argues that a separate, previously overlooked problem remains even after such corrections: accumulated FP8 quantization noise distorts the importance ratio, which in turn causes the worst tokens to be silently dropped from learning. The goal is to fix that distortion so full-pipeline FP8 RL trains as stably as BF16.

Key Contributions

1. Identifies a new source of FP8 RL instability. The authors show that full-pipeline FP8 RL suffers severe instability — anomalous mid-training entropy surges and garbled outputs — that is not resolved by existing train-inference mismatch corrections like TIS.

2. Traces the failure to compounded quantization noise in the importance ratio. The instability is attributed to FP8 noise accumulating and distorting the importance ratio, which disproportionately pushes negative-advantage tokens outside the trust region and incorrectly zeros their gradients.

3. Proposes Calibrated Clipping. A dynamic clipping method that aligns FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and rebalancing the upper bound accordingly.

4. Validates across algorithms, scales, and quantization granularities. Experiments span GRPO and DAPO, model sizes from 8B to 32B, and multiple FP8 scaling granularities, showing that entropy surges are eliminated and performance returns to a level comparable with the BF16 baseline.

Main Findings

• Instability persists after mismatch correction: Prior work focused on train-inference mismatch and correction techniques such as TIS; the authors report that full-pipeline FP8 RL still exhibits severe training instability regardless.

• The symptom is distinctive: Instability manifests as anomalous entropy surges occurring mid-training, together with garbled model outputs.

• The cause is compounded quantization noise: FP8 quantization noise accumulates and distorts the importance ratio used in the RL update.

• The distortion has a specific, harmful asymmetry: Negative-advantage tokens are disproportionately pushed outside the trust region, and their gradients are erroneously zeroed out.

• The consequence compounds over time: Because those tokens receive no penalty, pathological outputs are not properly corrected and instead accumulate as training proceeds.

• The proposed fix works across settings: Calibrated Clipping eliminates the entropy surges and restores performance comparable to the BF16 baseline across GRPO and DAPO, model scales from 8B to 32B, and multiple FP8 scaling granularities. The abstract does not report specific benchmark scores or numeric deltas.

Methodology in Plain English

The authors first run full-pipeline FP8 RL and observe that training breaks down partway through, even with existing mismatch corrections in place. Rather than attributing this to the training/inference gap again, they inspect the quantities used to decide how much each token's update should count — the importance ratio — and find that FP8's low precision introduces noise that stacks up, skewing that ratio.

The skew is not symmetric. Tokens whose advantage is negative (i.e., outputs the model should learn to avoid) are the ones most likely to be pushed outside the allowed trust region. When that happens, the standard clipping rule zeroes their gradient, so the model never learns from them — including from the garbled generations the training run is producing. Bad behavior therefore goes unpenalized and feeds on itself.

Their remedy, Calibrated Clipping, changes how the clipping bounds are set. Instead of using FP8-derived bounds as-is, the method dynamically adjusts them so they line up with what the bounds would have been in high-precision BF16. Concretely, it matches the lower clipping bound by quantile and rebalances the upper bound to stay consistent. The abstract presents this as the mechanism that both stops the entropy surges and brings accuracy back in line with the BF16 baseline.

Why This Matters

Impact on research: The paper reframes FP8 RL instability as a quantization-noise problem in the importance ratio rather than purely an engine-mismatch problem, which shifts where the community looks for fixes. It also implies that other low-precision RL pipelines may share the same failure mode, even if they use different correction techniques.

Real-world applications:

• RL post-training of reasoning models, where FP8 could reduce the cost of long training runs if stability can be preserved.
• Agentic LLM training, where unstable updates and garbled outputs are especially damaging because they can shape long-horizon behavior.
• Training infrastructure at scale, where FP8 throughput gains translate directly into reduced GPU-hours and cost for RL workloads.
• Model families across sizes, since the approach is reported to hold from 8B up to 32B, suggesting it is not tied to one scale.

Industry relevance: RL post-training is now a major cost center for frontier and mid-size model developers. If full-pipeline FP8 can be made as stable as BF16, teams gain a straightforward efficiency win without changing algorithms or giving up output quality — a practical concern for anyone running GRPO- or DAPO-style training in production.

Future Directions

• Formal analysis of quantization-noise accumulation in RL objectives. The abstract describes the distortion empirically; a principled account of how FP8 noise compounds across steps would help predict failure before it appears.

• Extension to other low-precision formats. The findings are specific to FP8. Whether the same importance-ratio distortion and clip-calibration fix apply to more aggressive formats is left open.

• Interaction with existing mismatch corrections. The paper positions Calibrated Clipping as addressing a separate problem from TIS-style corrections; how the two should be combined or tuned jointly is not resolved in the abstract.

• Broader algorithm and scale coverage. Validation covers GRPO and DAPO and 8B–32B models. Testing on other RL objectives, longer-horizon agentic training, and larger model scales remains an open question.

Target Audience

Machine learning systems and training-infrastructure researchers, RL post-training practitioners, and quantization specialists who work on efficient LLM training. Readers who already understand importance sampling, trust regions, and FP8 scaling will get the most from it; those new to low-precision RL training will need background reading first.

Authors’ abstract

Reinforcement learning (RL) has become a key technique for improving the reasoning and agentic abilities of large language models (LLMs). Although FP8 quantization can accelerate RL training, maintaining stability throughout an FP8 RL pipeline remains challenging. While previous works have focused on resolving train-inference mismatches using correction techniques like TIS, we reveal that full-pipeline FP8 RL still suffers from severe training instability, manifesting as anomalous mid-training entropy surges and garbled outputs. We trace this instability to a previously overlooked cause: compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients. As a result, pathological outputs are not properly penalized and accumulate over the course of training. To address this, we propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and rebalancing the upper bound accordingly. Extensive experiments across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities demonstrate that our approach successfully eliminates entropy surges and restores performance comparable to the BF16 baseline.