The Pulse
Score Centering Keeps LLM RL Stable at 30% Under INT4 Stress
A new Together AI paper identifies accumulated drift as a major cause of instability in large-language-model reinforcement learning. Its score-centering method reaches 30% training accuracy under severe sampler quantization, compared with 1
AI.info Team ·
Score centering reaches 30% training accuracy where truncated importance sampling reaches 12%, according to a new paper from Together AI researchers testing large-language-model reinforcement learning under severe quantization. The comparison uses an INT8 sampler with an INT4 key-value cache, a deliberately harsh setup designed to expose failures that may take much longer to appear under ordinary numerical differences.
The paper, “Score Centering Stabilizes Off-policy Reinforcement Learning,” was submitted to arXiv on September 17, 2026, by Martin Marek and Max Ryabinin. It argues that training often becomes unstable not simply because the rollout model differs from the training model, but because a small bias accumulates across repeated updates.
The 30% Result Comes From an INT4 Stress Test
Large language model reinforcement learning typically uses one system to generate sampled responses and another to calculate gradients. The two systems can differ because they use different numerical precisions, different kernels, separate software stacks, or model weights that are not synchronized at every step. The paper calls the resulting gap training-inference mismatch, or TIM.
Under ordinary policy-gradient training, the sampled response receives a reward and the model updates its token probabilities according to the response’s score. When the sampler and trainer match, the expected score is zero at each prefix. Under TIM, the paper shows, that expectation can become nonzero. The resulting drift pushes the trainer toward the sampler even when the reward contains no information favoring one sampled token over another.
Repeated synchronization can turn the problem into a feedback loop. A sampler running at lower precision or using older weights biases the trainer toward its own outputs; the updated trainer then supplies the next sampler checkpoint, carrying the bias forward. The authors describe the effect as a form of moving-teacher distillation that can compound until reward or accuracy collapses.
Subtracting the Sampler’s Expected Score
Score centering addresses that drift with an additive correction. For each token position, the method subtracts the expected score under the sampler’s distribution from the trainer’s score. The corrected score has zero expectation under the sampler, removing the drift term while preserving the reward-dependent covariance that supplies the learning signal.
The approach differs from importance sampling, which multiplies the gradient by a ratio comparing trainer and sampler probabilities. Those ratios can become very large for rare tokens, increasing gradient variance. Practical systems therefore clip or mask them, which limits the correction and can reintroduce bias.
Score centering uses no importance ratio, clipping threshold, or additional hyperparameter in its basic form. The correction is deterministic for a given prefix rather than dependent on which token happened to be sampled. The authors also show that the additive correction can be combined with importance sampling when the sampler is substantially stale.
A Top-128 Approximation Makes the Method Practical
Computing the exact correction would require retaining the sampler’s probability for every token in the vocabulary. The paper gives Qwen3’s vocabulary as roughly 152,000 tokens; storing full floating-point log probabilities for long rollouts would be prohibitively expensive.
The implementation instead records the sampler’s top 128 log probabilities and estimates the remaining tail using the trainer’s distribution, rescaled to match the sampler’s tail mass. Experiments reported in the paper find that using 128 entries, and even 32, performs on par with full score centering across the tested settings.
That design keeps the method compatible with rollout systems that discard most of the sampler’s output distribution after generation. The paper supplies a scalar-loss formulation and an implementation example in JAX, along with code linked from the arXiv page.
Results Across 0.6B and 30B Models
The experiments use Qwen3-0.6B-Instruct on the Countdown task and Qwen3-30B-A3B-Base on the mathematics portion of the INTELLECT-2 dataset. The researchers place every correction method on the same REINFORCE objective with group-centered rewards, using the same sampler, trainer, optimizer, advantages, and one stochastic-gradient step per batch.
For the 30B mixture-of-experts model, an FP8 sampler allows uncorrected policy gradient to train stably and reach 58% training accuracy in the reported run. Moving to an FP4 key-value cache causes uncorrected policy gradient to collapse within 200 steps, while masked importance sampling collapses later. Score centering remains stable at 52%, and truncated importance sampling reaches 51%.
The harshest setting combines an INT8 sampler with an INT4 key-value cache. Score centering reaches 30% training accuracy, truncated importance sampling reaches 12%, and every other method tested finishes below 5%. The result measures training accuracy under the specified stress test; it does not establish a held-out capability gain or an end-to-end speed improvement.
Stale Rollouts Still Need Importance Sampling
Quantization is not the only source of mismatch. The authors also test a sampler updated once every 64 training steps. Score centering alone performs less well in that setting because its remaining covariance is measured under the sampler’s distribution rather than the trainer’s.
Combining score centering with truncated or masked importance sampling performs best under severe staleness. Importance sampling partially corrects the difference in sampling distributions, while score centering removes the residual drift. The paper therefore recommends the combination when the trainer can move far from the sampler between synchronization points.
The authors are explicit about the limits of the evidence. Their headline comparisons use short sequences and intentionally severe mismatch so that differences emerge within hundreds of steps. They present those runs as a proxy for longer training with milder mismatch, not as a direct demonstration that the same accuracy gains will appear in production-scale, long-context systems.
The paper’s narrower claim is still concrete: under the tested conditions, canceling drift alone keeps reinforcement learning from collapsing where several importance-sampling and clipping methods fail. Whether the result transfers to ordinary quantization levels, longer agentic tasks, and less frequent but more realistic synchronization schedules remains an experimental question.