On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

AI summary

Overview

Research area: Reinforcement learning for large language models (LLM RL), specifically the systems-level and optimization-level discrepancy between the inference engine that samples responses and the trainer that computes gradients.

Technical level: Advanced. The paper is written for researchers who are comfortable with policy-gradient estimators, importance sampling, KL divergence, and eigenvalue arguments about training dynamics.

Scope: The paper gives an exact decomposition of the training-inference mismatch into a fixed numerical discrepancy and a self-induced policy lag, traces each through 40-step training runs on Qwen3-0.6B and Qwen3-1.7B, and derives a correction (DLC) from that account. Note that the supplied content is truncated inside Section 7.1, so the full derivation and the complete method description of DLC are only partially visible.

What This Paper Is About

On-policy RL for LLMs assumes the training responses come from the current policy, but in practice a separate inference engine (e.g., vLLM) samples them, so the data is subtly off-policy. This mismatch has two sources — the engine's numerical behavior and stale weights in asynchronous sampling — and the paper asks which property of a mismatch source determines which correction will work. The authors show that a single extra forward pass at the sampler's weights splits the error exactly into the two parts, that the two parts behave very differently during training, and that this difference explains why corrections that stabilize one source fail under the other.

Key Contributions

1. Exact, measurable split of the sampled-token error. Using the trainer's log-probability evaluated at the sampler's weights, the log-ratio error is decomposed without approximation into a numerical discrepancy (engine numerics) and a policy lag (stale weights). After centering removes the drift, the leftover bias is written as four named terms: prefix, continuation, token, and tail.

2. A feedback-loop account of the policy lag. The lag is generated by the trainer's own updates: the residual bias enters the next update, which enlarges the next lag. The authors identify the token and prefix terms as the parts that close this loop, locate the portion of known staleness bias that survives score centering, and show that the excess update norm of score centering over importance sampling has a positive quadratic term in the product of learning rate and step count.

3. An empirical validity range for importance weighting. Ratio weighting cuts the loop only while the lag ratio stays concentrated. The effective sample size of the ratio, not a Gaussian prediction, marks where this ends: on Qwen3-1.7B the crossing at one half occurs at step 11, three steps before the Gaussian prediction, and importance sampling leaves the noise floor at exactly that step.

4. The discrepancy-lag correction (DLC). DLC centers the numerical discrepancy, clips the lag ratio against the sampler's weights, and resynchronizes the sampler when the effective sample size of the lag ratio falls below one half — assigning each term of the bias to the layer (algorithm or infrastructure) that can actually remove it.

Main Findings

• The two mismatch sources have opposite dynamics. The quantized sampler's numerical discrepancy stays near 0.02 nats per token throughout training, while a stale sampler's numerical discrepancy is below 10⁻³ and its policy lag grows by two orders of magnitude on both model sizes. The two separate by step 5.

• A large constant offset does not by itself destabilize training. Under quantization the sequence offset stays at 8.9 nats on long responses, yet the residual bias stays at gradient noise. Under staleness, score centering leaves the floor near step 12, where the growing offset reaches the same size — so what matters is the covariance of the offset with the gradient the lag supplies.

• Corrections are source-dependent, reproducing prior reports. On Qwen3-0.6B under quantization, importance sampling reaches 0.74× the noise floor at step 40 and score centering 0.80×, both matching or near the 46.5% on-policy test accuracy, while the plain policy gradient reaches 12.7× and 26.4% accuracy. Under staleness, score centering reaches 31.7× the floor and 12.2% accuracy, whereas importance sampling with centering stays at 2.1× and 32.0%.

• The PPO reference policy explains a reported asymmetry. With the reference taken at the trainer before each batch, PPO reaches 699× the floor under staleness (worse than PPO without clip at 534×) and 14.3× under quantization. With the sampler's weights as reference, the lag sits inside the clipped ratio and the numerical discrepancy outside it, which is why the clip contains staleness but not quantization noise.

• The loop is governed by the product of learning rate and step count. Runs at half the learning rate fall on the same curves, and fitting the excess update norm gives a quadratic coefficient of c = 0.52 at R² = 0.987, versus 0.90 for a linear-only fit; the coefficient is positive in each of the six fits by learning rate and run.

• Score centering's residual bias is predictable and grows linearly. Measured bias norm grows linearly in the accumulated update with R² of 0.95 to 0.99 per run and reaches 2.5× the gradient noise by step 40, while importance sampling's bias stays within the noise. The predicted bias aligns with the measured one at a cosine of up to 0.55, against at most 0.07 for gradient noise.

• Lag concentrates on negative-advantage tokens under score centering. At step 20 the mean lag is −0.66 on negative-advantage tokens and −0.37 on positive-advantage ones, whereas importance sampling makes the two groups agree.

• Importance weighting fails at a measurable point. On Qwen3-1.7B the measured effective sample size agrees with the Gaussian value at step 10 and falls below one half at step 11, at a mean lag of −0.15 nats per token (the Gaussian would cross at −0.35). The deviation of importance sampling leaves the floor at that step and grows to 152× the floor by step 30.

• The lag becomes heavy-tailed. The 1% of tokens with the largest lag carry 78% of the summed lag at step 5 and 11% at step 40, and beyond step 20 the lag under score centering passes one nat, where the first-order loop analysis ends.

• DLC stays near the noise floor under staleness. On Qwen3-0.6B, DLC reaches 1.93× the floor at step 40 with 31.7% accuracy under staleness and 2.17× with 30.3% under combined quantization and staleness. On Qwen3-1.7B it stays at the noise floor through step 40 under a stale sampler, while importance sampling leaves the floor at step 11; resynchronization also prevents collapses that the clip alone leaves and that a KL measured on the sampler's prefixes does not detect.

Methodology in Plain English

The authors run controlled experiments where they know exactly what is wrong with the sampler. They build four samplers on top of a training setup: a reference sampler (vLLM in BF16 with CUDA graphs) that gives genuine on-policy training; a quantized sampler (FP8 weights in 128×128 blocks, dynamic activation quantization, FP8 KV cache) that isolates numerical discrepancy; a stale sampler (BF16, synchronized every 64 steps, so within 40 training steps it keeps the initial weights) that isolates policy lag; and a combined sampler that quantizes the stale weights. Crucially, they remove the confounds that live below the algorithm — the sampler reports the drawn distribution, token ids are passed to the trainer, and batching is outcome-independent — so only weight version and engine numerics reach the algorithm.

They train Qwen3-0.6B and Qwen3-1.7B on Countdown with ±1 rewards, 64 prompts, eight samples per step, SGD at learning rate 10⁻², three repeats of each configuration for 40 steps. The key measurement device is an extra trainer forward pass evaluated at the sampler's weights. Because that pass differs from the sampler by weights only and from the trainer by numerics only, the error splits exactly into lag and discrepancy — no approximation needed.

Deviation is measured as full-vocabulary KL to an on-policy model trained with the same configuration on prefixes drawn from the initial weights, averaged over the three runs and divided by a "noise floor": the mean KL between two independently trained on-policy models. That normalization turns every result into "how many times worse than irreducible run-to-run noise." They then instrument the three links of the feedback loop (lag growth, residual bias, bias accumulation), fit a quadratic in the learning-rate-times-step product, and use the second moment of the importance ratio to define when ratio weighting stops being valid. DLC is assembled from these measurements: one correction per term, each assigned to whichever layer can remove it.

Why This Matters

Impact on research. The paper replaces a vague notion of "training-inference mismatch" with two quantities that can be measured separately during a real run, and it gives a falsifiable criterion (effective sample size of the lag ratio below one half) for when a popular correction stops working. It also explains an otherwise puzzling asymmetry in prior work — that PPO survives staleness but collapses under quantization — through the position of the reference policy rather than through the clip's shape. The quadratic-in-(η·k) result reframes stability as a property of a feedback loop rather than of a single step.

Real-world applications:

• Asynchronous RL training pipelines (e.g., AReaL, PipelineRL) that deliberately keep sampling and training out of sync to raise throughput: DLC's resynchronization trigger gives a principled moment to stall the sampler instead of a hand-tuned interval.
• Quantized or low-precision inference for rollout generation, where FP8 weights and KV caches are used to cut cost: the finding that score centering alone nearly matches on-policy accuracy (46.5% versus 46.5%) tells practitioners the discrepancy is the benign source.
• Distributed training with tensor-parallel or kernel-level nondeterminism, where engine and trainer disagree numerically at identical weights: the paper's framing indicates centering addresses the leading bias without needing bitwise alignment.
• Long-running post-training jobs where collapse appears late: the ESS-based signal fires before divergence, and the paper notes that a KL measured on the sampler's prefixes does not detect the collapse that resynchronization prevents.

Industry relevance. Serving engines and trainers are typically different software stacks maintained by different teams; the split into "engine numerics" and "weight version" maps directly onto an ownership boundary between infrastructure and algorithm teams, which is exactly the division DLC's design uses.

Future Directions

• Extending the analysis past the first-order regime. The authors state that beyond step 20, when the lag exceeds one nat and spreads over the response, the first-order loop ends; what replaces it (and whether the ESS criterion remains the right trigger there) is left open.
• Choosing the resynchronization threshold beyond one half. The half-ESS trigger is calibrated on the two Qwen3 model sizes and on Countdown; whether it transfers to other model scales, tasks, and reward structures is untested in the reported content.
• Reconciling with the batch-reference PPO runs. The paper attributes PPO's staleness survival in prior work to the reference policy, but its own batch-reference runs reach 699× the floor; a direct study of the sampler-weight reference for PPO, beyond the shared clipped ratio in DLC, would close the gap.
• Applying DLC to the combined source at scale. DLC was run under staleness and under quantized-plus-stale, but the reported table shows no DLC run under quantization alone (where it reduces to score centering); a full grid at larger model sizes would test the claim that each term should be handled by its own layer.

Target Audience

Researchers and engineers working on RL post-training for LLMs — particularly those building or maintaining asynchronous rollout infrastructure, quantization for inference, or off-policy corrections such as truncated importance sampling and decoupled PPO. It will also interest theoretically inclined readers studying staleness bias and effective sample size in policy optimization. Beginners will find the setup and the Table 2 comparison accessible, but the bias derivations and eigenvalue arguments in Sections 5 and 6 require graduate-level familiarity with policy gradients.

Authors’ abstract

On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. \textit{We ask which property of a mismatch source decides the correction it needs.} Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a \textit{numerical discrepancy} of the engine and a \textit{policy lag} of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the \textbf{discrepancy-lag correction} (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.