Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for LLM mathematical reasoning, and the inference-time scaling that depends on policy diversity.

Technical level: Intermediate. Readers should be comfortable with pass@k metrics, PPO/GRPO policy-gradient training, and basic information-theoretic quantities such as entropy in nats.

One-sentence scope: Through exhaustive enumeration of the Countdown task's solution space, the paper decomposes RLVR's diversity loss into "access" (entering a valid solution family) and "execution" (computing within it), and shows the loss is concentrated almost entirely at the entrance.

What This Paper Is About

RLVR raises single-sample accuracy (pass@1) on reasoning tasks but shrinks the set of distinct valid solutions a model will produce, which limits the returns from repeated sampling, self-consistency, and verifier-guided search. Prior work measured this contraction only at the level of final answers or clustered traces, which cannot distinguish a model that never starts a valid solution path from one that starts it and then computes incorrectly. This paper builds a task whose entire solution space can be enumerated, splits it into "entrance families" defined by the first operand and operator, and measures which of the two failure modes actually drives the collapse.

Key Contributions

1. Access vs. execution decomposition. The authors introduce an exhaustive state-space decomposition on the Countdown task that separates solution initiation from downstream reasoning execution, using an exact symbolic solver to enumerate all canonical valid expressions.
2. Mechanistic localization of breadth loss. Across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct, they show the diversity collapse is concentrated at the entrance, with per-token likelihood shifts 11x to 16x larger before the first arithmetic operation than during subsequent reasoning.
3. Inference- and weight-space recovery. Parameter interpolation in late transformer layers (layers 20–28 blended with step-50 weights) and entrance-aware sampling recover up to 37% more solution coverage without sacrificing pass@1.
4. Benchmarking and scaling generalization. The entrance-narrowing signature recurs across 7B and 14B Qwen models on six standard math benchmarks, while staged SFT-to-DPO-to-RLVR pipelines preserve early-step calculation entropy.

Main Findings

• Accuracy and breadth trade off inversely. In the self-trained PPO run on Qwen2.5-3B, pass@1 rises more than fiftyfold between steps 50 and 275 while solution coverage at 320 samples falls 67%, from 0.337 to 0.111. The public GRPO series shows the same pattern from step 25 to step 450: accuracy more than triples while coverage drops 43%, from 0.467 to 0.268 at coverage@320.
• Opening diversity erodes fastest. In the GRPO run, entrance-family coverage among valid solutions falls from 0.932 to 0.571, and entrance Shannon entropy falls from 1.585 to 0.760 nats. Coarse operator-class coverage halves during training, and the finer first-evaluated-pair partition drops by two thirds to 0.117.
• The collapse is not just forgetting hard problems. On 58 problems solved at both step 50 and step 275, 64% remain solvable at step 275, but only 31% of the individual valid solution leaves found at step 50 are ever generated by the late policy. Mean coverage on this invariant subset falls from 0.564 to 0.286 even as formatting compliance approaches 100%. No problem missed at step 50 is newly unlocked at step 275, and at 2,048 samples per problem, 33 lost problems remain unrecovered.
• Likelihood shifts concentrate at the first operation. Under PPO, the entrance segment loses 4.50 nats per token (95% CI [4.37, 4.62]) versus 0.28 nats ([0.26, 0.30]) downstream, a 16x disparity. Under GRPO, the entrance loses 8.08 nats ([7.96, 8.19]) versus 0.71 ([0.58, 0.85]), an 11x ratio. The largest single token-level drop lands on the initial operand token that determines family selection.
• Supplying a minimal entrance revives "extinct" solutions. At PPO step 275, a solver-constructed prefix specifying only the first operand and operator raises designated-family completion from 0.018 to 0.212, matching what a fully completed first calculation achieves (0.276). The GRPO checkpoint shows the same recovery (0.104 to 0.188).
• Execution capability improves while access collapses. On 139 problems and 359 matched (problem, family, scaffold) tuples, conditional execution from identical supplied entrances rises by +0.107 (95% CI [0.073, 0.143]) between steps 50 and 275. For the 28 families observed at step 50 but never sampled at step 275 across 320 free rollouts, clamping the minimal entrance yields 0.529 completion at step 275, above the 0.475 baseline at step 50.
• Surface fixes fail; representational fixes work. On 100 held-out problems, prompt diversification moves coverage by only +0.014 ([0.000, 0.038]), forcing an alternative operator by -0.007 ([-0.017, 0.002]), and answer-only logit mixing by -0.006 ([-0.015, 0.000]). High-temperature decoding (T=1.0) expands coverage by +0.016 ([0.007, 0.027]) but drops pass@1 from 0.278 to 0.264.
• Layer interpolation recovers breadth at no accuracy cost. Blending layers 20–28 of the late checkpoint with step-50 weights raises coverage from 0.111 to 0.152, a 37% relative gain, with pass@1 at 0.305 versus the 0.278 control. Splitting the 64-sample budget evenly between steps 50 and 275 gives the largest coverage gain, +0.100 ([0.059, 0.147]), but shifts single-sample accuracy toward the early checkpoint (pass@1 0.161).
• Entrance-budgeted test-time compute helps exactly when it should. Uniformly distributing rollouts over solver-identified feasible entrances beats free resampling by +0.077 ([0.048, 0.109]) in coverage and adds +0.430 ([0.297, 0.570]) distinct entrance families across the 128 problems with non-trivial failure rollouts. At step 50, the same allocation yields no significant advantage.
• The phenomenon scales and generalizes. On six math benchmarks, SimpleRL raises macro pass@1 by 13% at 7B and 17% at 14B, but pass@64 barely moves (0.754 to 0.760 at 7B; unchanged at 0.774 at 14B), first-calculation entropy drops 30% at 7B and 41% at 14B, and the distinct-trace rate nearly halves.
• Longer horizons add a second bottleneck. When late RLVR policies are given first-calculation prefixes found by the base model but omitted under free RLVR sampling, they complete valid solutions at 0.850 on GSM8K and 0.713 on MATH500, but depth-controlled handoff experiments show execution variance growing with reasoning length.
• The tradeoff is a curriculum choice, not a law. An SFT model maintains more than double the solution coverage, and in the OLMo-3 7B ladder, pass@1 rises from 0.447 (SFT) to 0.508 (DPO) to 0.613 (RLVR) while first-calculation entropy stays stable (1.189, 1.269, 1.137) and distinct-trace rates stay near 0.77. DeepSeek-R1-Distill-Qwen-7B posts the highest pass@1 (0.526) and highest first-calculation entropy (1.240) among the evaluated 7B models.

Methodology in Plain English

The authors needed a task where they could see every possible correct answer, so they used Countdown: given three or four numbers and a target, produce an arithmetic expression using each number exactly once that hits the target. An exact symbolic solver enumerates every canonical valid expression, which lets the team group solutions by their opening move — the first operand and operator — into what they call "entrance families."

This grouping turns one question into two. Access asks whether the model ever chooses that opening move at all. Execution asks whether, once the opening move is fixed, the model can finish the arithmetic correctly. To measure access, they sample freely and count which families appear. To measure execution in isolation, they clamp a synthetic prefix onto the prompt — for example, "Let me try: 4 * " — which forces the family but leaves everything downstream unconstrained. These prefixes are constructed by the solver, not pulled from model traces, and a mutual-information check confirms they carry no information about which specific downstream solution will be reached.

They run this on two independent RLVR pipelines: their own PPO training of Qwen2.5-3B following TinyZero, using eleven checkpoints from step 25 to step 275 over 150 held-out solver-feasible problems, and the public Qwen-2.5-3B-R1-countdown GRPO series over 135 problems filtered to be disjoint from the 50,000-example training superset. To locate where probability mass moves, they use teacher forcing on identical reference solutions and split each trajectory at the first arithmetic operation, comparing per-token negative log-likelihood before and after that boundary.

Because standard math benchmarks have no enumerable solution space, they build proxy metrics on generated traces: the "first calculation" (earliest parsed arithmetic relation) and its categorical entropy, plus the "distinct-trace rate." They evaluate these across six benchmarks on 7B and 14B Qwen models, DeepSeek-R1-Distill-Qwen-7B, and the full OLMo-3 7B SFT/DPO/RLVR progression. Confidence intervals come from bootstrap resampling throughout.

Why This Matters

Impact on research. The paper reframes RLVR diversity loss from a vague "mode collapse" into a specific, measurable mechanism: policies stop entering valid solution branches rather than losing the ability to compute them. That distinction matters because it invalidates a whole class of remedies aimed at downstream reasoning quality and points instead at the opening decision. It also gives a mechanistic justification for two empirical tricks already in circulation — temporal checkpoint ensembling and weight interpolation — showing why they work.

Real-world applications (derived from the paper's findings):

• Test-time compute budgeting for reasoning models. The entrance-allocation result suggests systems that distribute a fixed rollout budget across known feasible strategies rather than resampling freely, especially where a problem's strategy space is enumerable or partially known.
• Model merging and checkpoint reuse. The layers 20–28 interpolation result offers a concrete recipe for recovering breadth from earlier checkpoints without retraining or losing accuracy.
• Training-pipeline design for reasoning products. The SFT-to-DPO-to-RLVR comparison indicates that teams can choose a curriculum that raises accuracy while retaining opening diversity, rather than accepting the tradeoff as inevitable.
• Diagnostics for deployed reasoning systems. The first-calculation entropy metric is cheap to compute on generated traces and could serve as an early warning that a trained model has narrowed its search.

Industry relevance. Any product relying on repeated sampling, majority voting, or verifier-guided search for math and reasoning — code assistants, tutoring systems, scientific computation tools — is bounded by how many distinct valid approaches the underlying policy can generate. This paper suggests the binding constraint is set at the very first token of computation and is partly recoverable after the fact, which is far cheaper than retraining.

Future Directions

• Extending the access/execution decomposition beyond enumerable tasks. The exact framework depends on Countdown's finite solution graph; only structural proxies are available for GSM8K, MATH500, and similar benchmarks. Better proxies, or domains with partial enumeration, would test how far the mechanism generalizes.
• Closing the long-horizon execution bottleneck. The authors show execution variance grows with downstream reasoning length, making it a compounding second bottleneck on open-domain problems. How to keep entrance breadth while also stabilizing long chains of computation remains open.
• Turning entrance-aware interventions into training objectives. The recovery methods are applied at inference or in weight space. Whether an RLVR objective can be modified — via reward shaping or exploration bonuses, as prior work attempts — to preserve entrance entropy during training is not settled here.
• Understanding why staged alignment avoids the collapse. SFT and SFT-to-D

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.