Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

AI summary

Overview

Research area: Post-training of large language models with reinforcement learning from verifiable rewards (RLVR), specifically the exploration problem in GRPO-style group-rollout training for mathematical reasoning.

Technical level: Advanced. The paper assumes familiarity with policy-gradient methods, group-relative advantages, PPO-style clipping, and embedding-level interventions in LLM decoding.

Scope: The paper introduces EDEN, a training-time method that perturbs one appended input-embedding position with anchored Gaussian noise for a fraction of each rollout group, and reports gains over GRPO and several exploration controls across three Qwen2.5 model scales and eight mathematical reasoning benchmarks.

What This Paper Is About

RLVR systems typically sample only a small group of rollouts (8–16) per prompt, and repeated sampling from the same policy often produces near-identical trajectories. When every rollout in a group gets the same reward, the group-relative advantage that GRPO depends on collapses to zero, so training stalls or the policy converges prematurely.

The authors ask whether a minimal change at the source of rollouts — perturbing the input embedding rather than the sampling temperature, the objective, or the policy weights — can produce trajectories the policy would rarely generate on its own, while keeping discrete decoding and test-time inference untouched. EDEN is their answer: a small fraction of each group is generated after appending a single noise-perturbed anchor embedding to the prompt.

Key Contributions

1. A rollout-source exploration method. EDEN replaces a small fraction of standard rollouts in each group with rollouts generated by the same policy conditioned on the prompt plus one appended, Gaussian-perturbed anchor embedding, using the same total rollout budget.

2. Directional exploration via noise-plus-anchor. Gaussian noise supplies local variation, while a single anchor embedding, shared by all perturbed rollouts of a prompt and drawn from a fixed bank of vocabulary-token embeddings, supplies a common conditioning direction. The design adds one embedding position, keeps discrete decoding, and changes neither model architecture nor inference.

3. Mechanisms for absorbing perturbed trajectories. Perturbed rollouts are selected at the batch level by keeping the top fraction p of absolute advantages, and their advantages are scaled by γ ∈ (0,1] before GRPO updates, with all trajectories evaluated under the clean prompt embedding.

4. Empirical validation across scales and controls. EDEN is compared against GRPO and against same-pipeline exploration controls (higher sampling temperature, larger entropy bonus, asymmetric clipping, larger rollout groups) on three Qwen2.5 models, with additional analysis of training dynamics and ablations on noise, anchoring, bank size, and noisy-rollout fraction.

Main Findings

• Average gains at 1.5B. On Qwen2.5-Math-1.5B, EDEN reaches an average Pass@1 of 61.34 versus 58.90 for GRPO (+2.44) and an average Pass@8 of 73.96 versus 70.90 (+3.06). The same-pipeline controls land lower: Higher Temperature (T=1.2) at 58.97 Pass@1 / 70.69 Pass@8, and Clip Higher at 59.29 Pass@1 / 71.87 Pass@8.

• Average gains at 7B. On Qwen2.5-Math-7B, EDEN averages 65.48 Pass@1 versus 64.87 for GRPO, and 78.01 Pass@8 versus 74.37. Pass@8 improves on MATH (93.00 vs 88.80), GSM8K (97.04 vs 96.21), Olympiad (64.39 vs 58.31), Gaokao (80.52 vs 77.14) and Minerva (50.00 vs 42.65), while AMC is identical (83.13).

• Coverage persists at large sampling budgets. Pass@256 on AIME improves over GRPO by 6.67 points on average across model scales, including a 13.33 point gain on AIME'25 with Qwen2.5-Math-7B (66.67 vs 53.33). On Qwen2.5-Math-1.5B the AIME average rises from 51.67 to 58.33.

• More rollouts alone do not replicate the effect. Raising GRPO's budget from 8 to 32 rollouts per prompt improves Pass@1 by only 0.5 points and slightly reduces Pass@8, whereas EDEN at 16 rollouts reaches 47.95 Pass@1 / 69.01 Pass@8 on Qwen2.5-0.5B versus 45.78 / 66.21 for GRPO at 32.

• Fewer dead groups, higher entropy. From about step 750 onward EDEN has fewer zero-advantage groups, reaching 0.53 versus 0.58 for GRPO at step 2000 on Qwen2.5-Math-1.5B, and it maintains higher training entropy than every GRPO configuration after warm-up.

• Diversity appears before any update. Even before RL training, correct anchor-perturbed trajectories spread more widely than standard rollouts in aligned 4-gram space, with an RMS radius about 1.6 times as large and more samples outside the standard-rollout hull.

• Anchor and noise are both necessary. On Qwen2.5-0.5B (averaged over MATH500 and GSM8K), EDEN scores 47.95 Pass@1 / 69.01 Pass@8; removing noise drops to 45.42 / 66.51 (GRPO level), norm-matched random noise without an anchor gives 43.39 / 65.45, a new random anchor per rollout gives 43.86 / 65.81, and a single fixed random anchor gives 46.37 / 67.78.

• Bank size helps modestly. On Qwen2.5-Math-1.5B, K=1 already yields 48.45 Pass@1 / 61.96 Pass@8 versus GRPO's 48.34 / 59.43; K=4 gives 48.58 / 62.02 and K=8 gives 49.33 / 62.13.

• A modest noisy fraction is best. Comparing 0% (GRPO), 25% (12 standard + 4 noisy) and 50% (8 standard + 8 noisy) on Qwen2.5-0.5B, the 25% setting performs best while 50% matches GRPO.

• Filtering stabilizes training. Removing the advantage-based filter causes unstable updates with frequent gradient-norm spikes from high-variance noisy rollouts; the filtered version scores 47.03 average versus 45.62 without it.

• Conventional controls fall short. Raising the entropy bonus on Qwen2.5-Math-1.5B increases policy entropy throughout training without improving validation accuracy, which declines at larger bonuses; higher temperatures give inconsistent gains and extreme values destabilize training. A temperature warmup without embedding perturbations shows the entropy collapse returning, so the schedule alone does not explain EDEN's sustained exploration.

Methodology in Plain English

The starting point is GRPO. For each prompt the policy samples a group of rollouts (16 by default), scores them with a binary verifiable reward (1 for correct, 0 otherwise), and standardizes rewards within the group to get advantages that drive the update.

EDEN changes only how part of that group is generated. For each prompt it splits the 16 rollouts into 12 standard and 4 noise-perturbed ones. For the perturbed subset, the system takes one anchor embedding — a whole vocabulary token's embedding vector, drawn from a fixed bank of 8 candidates built once before training from the model's own vocabulary and screened for viability — and appends it as one extra position after the prompt. Each of the 4 rollouts gets its own independent isotropic Gaussian noise added to that shared anchor, scaled relative to the anchor's norm by ρ (0.12 for Qwen2.5-0.5B, 0.18 for the math models). The policy then generates normally with discrete decoding from this slightly shifted conditioning. Because the perturbation sits before the first generated token, autoregressive conditioning carries its influence through the whole response without any intervention during decoding.

Exploration strength is controlled on three fronts: the anchor is resampled per prompt (so different groups explore different directions), the noisy rollouts' sampling temperature follows a cosine schedule from 0.2 to 1.0 over a warmup (250 steps for 0.5B, 200 for 1.5B, 400 for 7B) to avoid early instability, and a single-anchor-per-group design keeps variation structured rather than scattered.

Rewards and group-relative advantages are computed jointly over all 16 rollouts. Perturbed rollouts are then filtered globally across the prompt batch: only those whose absolute advantage falls in the top p = 0.5 fraction are kept, so a prompt may retain anywhere from 0 to 4 of them. Under binary rewards the minority outcome carries the largest advantage magnitude, so this filter preferentially keeps trajectories that actually change the group signal. Kept perturbed trajectories get their advantages scaled by γ = 0.5, and the policy update removes the appended position and evaluates every trajectory's token log-probabilities under the clean prompt — no importance-sampling correction is used for this generation/update mismatch. Optimization otherwise follows GRPO with asymmetric clipping (ε_low = 0.2, ε_high = 0.28), β = 0 for the KL term, and an entropy coefficient α = 0.001.

Training used the verl framework with vLLM for generation, batch size 128, learning rate 1×10⁻⁶, roughly 400 steps (6 epochs) for Qwen2.5-0.5B and about 2000 steps (10 epochs) for the math models, on 8× H100 GPUs. Pass@8 and Pass@256 evaluations used temperature 0.6 and top-p 0.95.

Why This Matters

Impact on research. The paper reframes RLVR exploration as a question about rollout conditioning rather than rollout count, sampling temperature, or the objective. Its ablations show that a shared anchor and per-rollout noise play complementary roles, and that GRPO's entropy collapse is not fixed by drawing more samples from the same distribution — a result that bears directly on the dynamic-sampling and entropy-regularization lines of work. The intervention is orthogonal to architecture and inference, so it can be stacked on existing GRPO variants and clipping schemes.

Potential real-world applications:

• Post-training math and science reasoning tutors where the model must find multiple solution paths, not just one memorized route.
• Code generation and verification pipelines, where a verifier can check outputs automatically and diverse correct programs matter.
• Any verifiable-reward domain with sparse binary outcomes (proof assistants, formal specification repair, structured extraction with checker-validated output).
• Cost-constrained training runs where rollout generation dominates compute and increasing group size is not affordable.

Industry relevance. EDEN adds no model parameters, no extra decoding machinery, and no inference-time cost; it reuses the existing rollout budget and integrates with existing GRPO training stacks. That makes it a low-friction drop-in for teams already running GRPO/DAPO-style post-training, and the "more rollouts is not enough" finding is directly actionable for anyone budgeting GPU hours toward larger group sizes.

Future Directions

• Learnable or adaptive anchor banks. The authors note the current bank is fixed and constructed offline; dynamic selection or joint optimization of anchors with the policy could target exploration more precisely.
• Structured perturbation geometry. Gaussian noise centered at anchors could be replaced by anisotropic noise or task-dependent directions to cover trajectory space more effectively.
• Better handling of the generation/update mismatch. EDEN currently relies on advantage filtering plus γ-scaling instead of an importance-sampling correction for trajectories generated under perturbed but optimized under clean conditioning; a principled correction is an open design question.
• Interaction with other RLVR levers. Since EDEN is compatible with existing refinements (dynamic sampling, token-level losses, alternative clipping), how far its gains compound with each of them — and whether the optimal noisy fraction shifts with model scale and task difficulty — remains untested beyond the reported settings.

Target Audience

Researchers and engineers working on LLM post-training and reinforcement learning, particularly those implementing GRPO/DAPO-style pipelines who are hitting entropy collapse or flat pass@k curves. It is most useful to readers already comfortable with policy-gradient mechanics and embedding-level model access; the paper is not an introductory treatment of RLVR. Practitioners focused on reasoning-benchmark performance with fixed compute budgets will find the ablations on noisy-rollout fraction, bank size, and rollout budget the most directly transferable parts.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.