Decoupling Exploration from Optimization in RLVR

AI summary

Overview

Research area: Reinforcement learning for large language model reasoning — specifically reinforcement learning with verifiable rewards (RLVR), and how exploration is incentivized inside such training runs.

Technical level: Advanced. The paper assumes familiarity with policy-gradient RL for LLMs (GRPO, DAPO, clip-higher, entropy collapse), reward shaping with intrinsic novelty bonuses, and reasoning-trace distillation.

Scope in one sentence: The paper proposes Exploration-Distillation (ExpDis), a training framework that moves novelty-driven exploration into separate explorer policies whose filtered trajectories are distilled into a student policy trained only for correctness, and shows it outperforms DAPO at matched wall-clock budget on seven mathematical reasoning benchmarks across the Qwen3 and Ministral model families.

What This Paper Is About

RLVR improves reasoning by sampling solutions, scoring them with a programmatic verifier, and shifting probability toward those that pass — so it can only reinforce what the model already samples. Adding an explicit novelty bonus to the reward is the obvious way to encourage new strategies, but the paper argues the reward supervises only a narrow slice of the model's knowledge, so damage caused by exploration outside that slice cannot be detected or repaired. The paper's goal is to make aggressive exploration safe by giving exploration and optimization to two different policies, communicating only through filtered trajectories rather than shared weights.

Key Contributions

1. A decoupled RLVR framework (ExpDis). Exploration and optimization are assigned to separate model instances trained from the same base checkpoint: an explorer policy trained with a novelty bonus, and a student policy that never sees the novelty term and is trained with standard correctness-only RLVR.
2. Two independent scaling axes. The framework scales in breadth (K parallel explorer policies whose pooled trajectories are filtered and distilled into one student) and in depth (R rounds of alternating exploration and distillation, each round initialized from the previous student and drawing prompts from its own non-overlapping shard), with the novelty weight annealed from 0.75 to 0.25 over four rounds.
3. Empirical results across models and benchmarks. ExpDis outperforms DAPO at the same wall-clock budget on Qwen3-1.7B, Qwen3-4B, and Ministral-3-3B-Instruct-2512, with improved pass@k scaling, and avoids the general-capability degradation seen when a novelty bonus is added directly to DAPO.
4. Analysis and ablations. The paper reports diversity metrics, an analysis of how explorer and student policies evolve across rounds, ablations of the filtering stage and of alternative novelty bonuses (kNN and elliptical), and releases all code and checkpoints.

Main Findings

• ExpDis beats DAPO under matched compute. Mean pass@k over the five primary benchmarks (Table 4): on Qwen3-1.7B, ExpDis reaches 51.81 at pass@1 and 83.04 at pass@64 versus DAPO's 45.63 and 74.64; on Qwen3-4B, 66.50 and 87.63 versus 62.58 and 81.14; on Ministral-3-3B, 34.76 and 72.56 versus 29.63 and 60.12.
• A single round already beats a 4× longer DAPO run. ExpDis single-round pass@1 is 48.93 versus DAPO trained for 4 times as many steps at 47.28 on Qwen3-1.7B, 64.69 versus 63.64 on Qwen3-4B, and 32.16 versus 30.90 on Ministral-3-3B. The corresponding pass@64 values are 77.83 versus 76.24, 84.50 versus 82.82, and 64.59 versus 62.36.
• Adding the novelty bonus directly to DAPO hurts. DAPO (+novelty) scores below plain DAPO on the five primary benchmarks (45.08 versus 45.63, 62.20 versus 62.58, 29.21 versus 29.63 at pass@1) and degrades below the base model on prior-capability benchmarks even while improving on math.
• Depth contributes more to pass@1 than breadth. Holding the budget fixed at 300 RL updates on Qwen3-1.7B (Figure 4), adding rounds improves pass@1 more than adding parallel explorers, while combining both gives the largest gains in pass@1 and pass@64. At this budget, gains saturate and show signs of regression after three explorers and four rounds.
• Exploration can be pushed much harder than a single policy could tolerate. The explorer's diversity can rise far beyond DAPO (+novelty) without downstream reasoning degradation, because trajectories are filtered for correctness and quality before distillation. At lambda = 0.5, the explorer's InterDistinct-4 is 0.422 and AnswerDistinct@n is 12.61, while the student reaches 0.357 and 9.93.
• Pushing the novelty weight too far does eventually hurt. At lambda = 2.0 the explorer reaches InterDistinct-4 0.550 and AnswerDistinct@n 23.61, but the student falls to 0.283 and 7.87, below the DAPO student's 0.308 and 8.50. Performance roughly peaks at lambda = 0.5, and several values of lambda outperform DAPO.
• Student policies are more diverse than DAPO. On AIME24 generations for Qwen3-1.7B, ExpDis shows token entropy 0.391, semantic diversity 0.143, and InterDistinct-4 0.339, versus DAPO's 0.251, 0.096, and 0.308; DAPO trained 4 times as long shows 0.265, 0.091, and 0.302.
• Both stages of the pipeline are necessary. On Qwen3-1.7B, distilling unfiltered explorer traces and then running RL reaches 49.79% on AIME24, and distilling filtered traces without subsequent RL reaches 49.84%; DAPO reaches 50.05%, and only filtering followed by RL reaches 52.76%.
• Quality filtering on top of correctness filtering helps. With three explorers, keeping every correct trajectory (NaivePool) gives a mean of 48.95%, no better than a single explorer at 48.93%; applying the quality filter raises this to 49.47%. Keeping only the shortest valid trajectory per problem outperforms keeping every correct trajectory.
• Gains are not specific to RND. Replacing RND with a kNN-based bonus (mean 47.37) or an elliptical bonus adapted from prior work (mean 48.06) drops mean accuracy by at most 1.6 points relative to RND's 48.93, and both still outperform DAPO's 45.63.
• Reasoning faithfulness declines mildly under the novelty bonus. Using the faithfulness definition from prior work, the novelty bonus typically causes faithfulness to decline, though the impact is mild by the final round (averaged over three models).

Methodology in Plain English

The framework runs in repeating rounds. In each round, the researchers take the current best model and train one or more explorer policies on top of it using DAPO, but with an extra novelty term added to the reward. The novelty signal is a Random Network Distillation bonus: small predictor networks try to match frozen random target networks on features pulled from intermediate layers of the language model's own hidden states, and the prediction error is used as the measure of novelty. Crucially, this bonus is only credited on trajectories the verifier marks correct, so a model cannot farm reward by generating novel nonsense.

The explorers are then used as a source of data, never as the deployed model. Their trajectories are filtered: a trajectory survives only if its final boxed answer is correct, it terminates within a fixed token budget, and it contains no repetition loops. From the survivors, only a single shortest trajectory per problem is kept, and the resulting set is capped at 500 trajectories per round. The current model is then supervised-fine-tuned on those trajectories with masked cross-entropy loss (2 epochs), and finally trained with standard DAPO using a correctness-only reward and no novelty bonus. That final model is the student, which becomes the starting point for the next round.

Scaling works along two axes. Breadth means training K explorers in parallel with the same novelty weight and pooling their filtered trajectories. Depth means running more rounds, with each round drawing prompts from its own non-overlapping shard so that a stronger initialization meets prompts it has not yet trained on, and with the novelty weight annealed downward. Each ExpDis run spends 200 RL updates on explorers and 100 on the student, against 300 for single-model baselines (and 1,200 total for the 4× DAPO run); parallel explorers run concurrently, so wall-clock time is equal across configurations.

Why This Matters

Impact on research. The paper reframes a known failure mode of novelty-driven RLVR not as a problem with novelty as a signal, but as a problem with where the novelty bonus is applied. That reframing suggests a general design pattern — let a throwaway policy be aggressive, and let a separate deployed policy inherit only its verified output — which the authors argue is compatible with any exploration mechanism, including ones too destabilizing to apply to a trained model directly. It also connects LLM RLVR to a long lineage of decoupled exploration in classical RL.

Real-world applications (as framed by the paper's discussion and results).

• Mathematical reasoning assistants, where the reported gains are measured across AIME24, AIME25, AIME26, MATH500, AMC23, Minerva-Math, and GSM8K.
• Long-horizon agentic tasks, where the authors suggest the problem of rare or unfamiliar correct solutions may be especially acute.
• Theorem proving, which the paper names as a domain where a correctness-only objective may avoid sampling genuinely new strategies.
• Open-ended scientific problems, also named by the authors as a setting where aggressive exploration may be necessary for discovering new ideas.

Industry relevance. The experiments are compute-matched to DAPO, and parallel explorers add no wall-clock time over a single explorer, so the method is presented as a way to get more out of a fixed training budget rather than as a way to spend more. That matters for teams running RLVR pipelines who cannot afford to degrade a deployed model's general capabilities — knowledge (MMLU-Pro, MMLU-Redux, GPQA-Diamond), logical reasoning (ZebraLogic), and instruction following (IFEval) — in exchange for math gains. The reported hardware is a single TPU v5litepod-64 slice, and the authors release code and checkpoints.

Future Directions

• Explorer specialization. Because an explorer only has to produce useful traces, the best exploration method under ExpDis may look quite different from the best one under standard RLVR. Explorers could use different novelty signals or sampling strategies to specialize in different parts of the solution distribution, or use architectures other than the student's (the paper's explorers share the student's architecture).
• More aggressive exploration mechanisms. The authors speculate that exploration methods previously too destabilizing to apply to a trained model may become usable once exploration is decoupled, and that ExpDis may unlock new and more aggressive mechanisms.
• Revisiting the budget optimum at scale. The observed saturation after three explorers and four rounds is measured at 300 RL updates; the authors expect the optimal number of rounds and explorers to increase as the budget grows.
• Extending beyond mathematical reasoning. The paper argues the value of decoupling should grow as models become proficient at strategies they have already seen, and flags long-horizon agentic tasks, theorem proving, and open-ended scientific problems as settings where this could matter most.

Target Audience

Researchers and engineers working on RLVR and post-training for reasoning models, particularly those already familiar with GRPO, DAPO, and Dr. GRPO and interested in exploration, reward shaping, or mitigating capability degradation. It is also relevant to practitioners who must defend general capabilities while pushing reasoning performance under a fixed compute budget, and to readers interested in the analogy the authors draw between decoupled exploration in classical RL and exploration in LLM training.

Authors’ abstract

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with strong novelty incentives has seen limited success and can degrade model quality. Because verifiable rewards supervise only a narrow slice of the model's knowledge and behavior, such degradations are difficult to recover from. Instead, we decouple exploration from optimization in a framework we call Exploration-Distillation (ExpDis). We train one or more explorer policies with a novelty bonus in the reward, filter their trajectories for correctness and quality, and distill them into a separate student policy. The student policy is then trained without a novelty bonus. We repeat the above procedure for several rounds, alternating between exploration and optimization. This decoupling allows us to aggressively scale exploration without degrading the student policy. Across seven mathematical reasoning benchmarks and two model families, ExpDis outperforms DAPO at the same wall-clock budget. Moreover, we observe improved pass@$k$ scaling, indicating that ExpDis produces models that generate more diverse correct solutions.