Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification

AI summary

Overview

Research area: Reinforcement learning for large language model reasoning, specifically Reinforcement Learning with Verifiable Rewards (RLVR) and the design of better rollout/sampling strategies.

Technical level: Advanced. The paper assumes familiarity with GRPO-style policy optimization, token-level log-probabilities, importance ratios, and KL-regularized PPO-style objectives.

Scope: The paper introduces an online selective rollout framework called Lens (Less Noise Sampling Framework) that identifies and removes "interference tokens" from low-success prompts and transfers the resulting successful rollouts back to supervise training on the original, noisy prompts.

Authors and venue context: Yiju Guo and Yankai Lin (Gaoling School of Artificial Intelligence, Renmin University of China), Tianyi Hu (Department of Computer Science, Aarhus University), and Zexu Sun (Baidu Inc.). Posted as arXiv:2601.21244v3 [cs.LG], published 2026-01-29, listed under Machine Learning. Code is linked at the repository RUCBM/LENS.

What This Paper Is About

RLVR methods such as GRPO depend on sampling at least some correct rollouts to produce useful learning signals, but on hard reasoning tasks correct rollouts are rare, so training becomes unstable or collapses. The authors find that many of these exploration failures are not caused by genuine problem difficulty, but by a small number of prompt tokens that introduce interference. Their goal is to identify and remove those tokens so that more rollouts succeed, and then to use the successful rollouts to teach the model to ignore the same interference when working with the original, unmodified prompt.

Key Contributions

1. A token-level diagnosis of exploration failure. Through fine-grained token-level analysis, the authors show that many failures come from a few tokens (fewer than 5%) that cause excessive interference, and that simply pruning these tokens improves rollout accuracy on previously failed DeepMath samples by over 20% across all model families (measured as Average@8).

2. The Interference Score and a purification mechanism. They define the interference score of a token as the absolute difference between the current policy's log-probability and the reference model's log-probability for that token given its prefix. Tokens are ranked by this score, and the top k = ⌈γ · |x_i|⌉ tokens are deleted, with γ set to a small value such as 1% to 5%.

3. Calibrated Rollout Policy Optimization (CRPO). Because removing tokens only improves rollout accuracy for about 20% of prompts, the framework does not simply train on denoised prompts. Instead, when the original prompt's success rate falls below a threshold τ and the denoised prompt achieves higher accuracy, successful rollouts from the denoised prompt replace a subset of failed rollouts from the original prompt and are used as supervision on the original prompt through a reweighted, importance-corrected, PPO-style clipped objective.

4. Empirical evidence of a better performance–efficiency trade-off. Lens reports a 3.88% average gain with over 1.6× speedup on math reasoning, plus a 1.83% gain on scientific and general reasoning, while outperforming both scaling-exploration and prompt-filtering baselines under a strictly lower computational budget (no increase in rollout count or training epochs).

Main Findings

• Interference is concentrated in very few tokens. The token-level interference score distribution shows only a few tokens with high interference, and those tokens make up fewer than 5% of prompt tokens.

• Pruning alone already helps rollout success. Removing interference tokens raised average rollout accuracy on previously failed DeepMath samples by more than 20% across all model families, but it improved rollout accuracy for only about 20% of prompts — the motivation for the calibration stage rather than plain filtering.

• Stronger math reasoning across model families. On seven math benchmarks, Lens applied on top of GRPO reaches an average of 27.03 for Llama3.2-3B-Instruct (vs. 23.98 for GRPO), 49.15 for Qwen3-4B-Base (vs. 44.41), and 53.36 for Qwen3-8B-Base (vs. 49.52). On Qwen2.5-3B it reaches 33.43 (vs. 30.51) and on Qwen2.5-7B, 43.88 (vs. 40.79). Averaged across the seven math benchmarks, the reported gain is 3.88% with over 1.6× speedup.

• Outperforms rollout-scaling and prompt-filtering baselines. Lens beats GRPO_extended (which doubles the rollout budget per instruction) and also surpasses DAPO and GRESO, the post-rollout and pre-rollout zero-variance filtering methods, even when those baselines are trained for 2× more rollouts or epochs in their extended variants. The authors attribute this to filtering strategies discarding exploration on challenging samples, which is consistent with Lens showing larger gains on hard benchmarks such as AMC23 and AIME24.

• Generalization beyond math. Models trained only on mathematical data were evaluated on SuperGPQA, GPQA-Diamond, BBEH, and MMLU-Pro. Lens outperforms GRPO on both the Qwen3 and Llama3.2 families, with an average gain of 1.83% — for example, 42.35 vs. 39.94 on Qwen3-8B-Base and 24.19 vs. 22.97 on Llama3.2-3B-Instruct.

• Faster convergence. Lens reaches the peak average accuracy of the GRPO baseline using 1.67× fewer gradient steps on MATH-500 and 1.64× fewer on OlympiadBench.

• Healthier training dynamics. Sampling accuracy distributions show Lens substantially reduces the proportion of zero-reward (Failure) prompts in early (steps 1–100), middle (steps 101–200), and late (steps 201–300) training stages, shifting more prompts into the Mid and High categories. Accuracy and response length rise faster while entropy stays moderate, with earlier emergence of long-form reasoning.

• Threshold sensitivity. Testing τ ∈ {0.125, 0.25, 0.375, 0.5}, the authors report τ = 0.125 concentrates the signal on the hardest samples near the capability frontier (favoring Minerva and AMC23), while τ = 0.5 gives broader coverage of medium- and high-difficulty samples and the best aggregate performance (33.43 average), so τ = 0.5 is the default.

• Pruning-ratio sensitivity tracks model capacity. Across pruning thresholds from 1% to 5% on Qwen2.5-3B and Qwen2.5-7B, Lens matches or exceeds GRPO at all thresholds; the weaker 3B model performs best with a higher threshold while the stronger 7B model performs best with a lower one, suggesting limited-capacity models are more affected by interference tokens.

• The pruning strategy matters. Compared against resampling, random pruning, and gradient-based pruning under the same setup and pruning ratio, Lens gives the best average (33.43), ahead of resampling (32.42), gradient-based pruning (32.39), random pruning (32.14), and GRPO (30.51).

• Compute overhead. On 8× NVIDIA A800 GPUs using the verl framework, Lens costs 1.27× to 1.62× the step time of standard GRPO (G = 8), still below the brute-force G = 16 setting. Average step times are 333 s vs. 214 s for GRPO on Qwen2.5-3B, 463 s vs. 344 s on Qwen2.5-7B, 475 s vs. 331 s on Qwen3-4B-Base, 726 s vs. 477 s on Qwen3-8B-Base, and 291 s vs. 229 s on Llama-3.2-3B-Instruct.

• Semantic preservation. BERTScore between original and pruned prompts exceeds 0.98 F1 across three backbones (0.9875 for Qwen3-4B, 0.9871 for Qwen3-8B, 0.9877 for Llama3.1-8B, as labeled in the paper), with precision around 0.988 and recall around 0.987. The authors also argue functional equivalence: since rewards are computed against a unique ground-truth answer, post-pruning accuracy gains imply the pruned prompts remain semantically equivalent.

Methodology in Plain English

The starting observation is that a model in RLVR training sometimes fails a problem not because the problem is hard but because a handful of words in the prompt derail its reasoning. The authors want to find those words automatically.

They measure interference with a simple quantity: for each token in a prompt, they compare how likely the currently trained model thinks that token is versus how likely a frozen reference model thinks it is. A big disagreement suggests the trained model has drifted away from the reference distribution in a way that hurts exploration. Tokens with the largest disagreements are labeled interference tokens.

During training, the method samples a group of rollouts for each prompt as usual. If a prompt's success rate is below a threshold τ, the system computes interference scores for its tokens, removes the top k = ⌈γ · |x_i|⌉ tokens (with γ kept small, 1%–5%, to preserve meaning), and samples another group of rollouts from this cleaned prompt. If the cleaned prompt yields a higher success rate, the successful rollouts from it replace some of the failed rollouts from the original prompt. Each rollout is then reweighted depending on its source, corrected for the fact that some rollouts were generated from a different prompt, and optimized with a PPO-style clipped objective plus a KL penalty against the reference model. In effect, the model is trained on the original noisy prompt but receives signal from rollouts that only succeeded after the noise was removed, teaching it to disregard the interfering tokens.

Experiments train Llama-3.2-3B-Instruct, Qwen2.5-3B, Qwen2.5-7B, Qwen3-4B-Base, and Qwen3-8B-Base on Openr1-Math-46k, evaluating on MATH500, AMC23, AIME24, AIME25, GaokaoEN-2023, Minerva, and OlympiadBench, plus out-of-domain tests on SuperGPQA, GPQA-Diamond, BBEH, and MMLU-Pro. Baselines include GRPO, GRPO_extended, DAPO, DAPO_extended, GRESO, and GRESO_extended. Training uses a learning rate of 1 × 10⁻⁶, rollout and update batch sizes of 128, 8 rollouts, temperature 1, top-p 1, and maximum response length 4096 tokens; evaluation uses temperature and top-p of 1.0 with a maximum generation length of 4,096 tokens, reporting Pass@1 accuracy and averages over 16 generation samples for AMC23, AIME24, and AIME25.

Why This Matters

This work reframes a common RLVR failure mode: rather than treating zero-reward prompts as unusable and throwing them away, or spending more compute on more rollouts, it argues that such prompts often contain a recoverable learning signal hidden behind a small amount of prompt-level noise. That matters for research because it points at the input side of the pipeline — instruction tokens — rather than only the output side, which is where most prior credit-assignment work concentrates.

Real-world applications:

• Training reasoning models in domains where labeled data is scarce and correct answers are verifiable, such as mathematics and formal problem solving.
• Scientific question answering and graduate-level science benchmarks, where the paper shows transferable gains without domain-specific training.
• Robust deployment of assistants that receive messy, templated, or verbosely formatted user prompts containing distractors.
• Cost-constrained training pipelines, where reaching a target accuracy in fewer gradient steps directly reduces GPU spend.

Industry relevance: The method is described as plug-and-play on top of GRPO and was validated with the verl framework on 8× NVIDIA A800 GPUs, with a reported overhead of 1.27× to 1.62× per step that remains cheaper than simply doubling the sample group size. One author's internship at Baidu and the Baidu affiliation suggest direct industrial interest in making RL post-training cheaper for reasoning models.

Future Directions

• Scaling to larger models. The authors state explicitly that experiments were limited to models up to 8B parameters and that evaluating 32B or 70B models remains future work.
• Beyond binary rewards. Effectiveness was validated primarily on tasks with binary rewards; applicability to environments with multi-dimensional scoring is listed as requiring further investigation.
• Combining with other GRPO variants. The method was demonstrated within the GRPO framework, and the authors have not yet explored combining it with algorithms that optimize rollout frequency or reward functions.
• Understanding what makes a token an interference token. Since pruning only improved accuracy for about 20% of prompts and the optimal pruning ratio and success-rate threshold both varied with model capacity, the conditions under which purification helps remain an open question.

Target Audience

This paper is most useful for researchers and engineers working on reinforcement learning post-training for language models, especially those dealing with sparse or collapsing reward signals in GRPO-style pipelines. It will also interest practitioners who need to reduce the compute cost of reasoning-model training, and readers familiar with credit assignment, rollout sampling, and policy optimization who want a concrete, token-level account of why exploration fails and how small prompt edits can change it. Readers without a background in RL fine-tuning will find the high-level idea accessible, but the algorithm itself requires comfort with importance ratios, group-relative advantages, and KL-regularized objectives.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6$\times$ speedup on math reasoning, and a 1.83% gain on scientific and general reasoning. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.