What Outcome RL Buys: Finishing, Reliability, Rarely Reach

AI summary

Overview

Research area: Natural language processing, specifically reinforcement learning (RL) with verifiable rewards for reasoning language models, and evaluation of what such training actually changes. Technical level: Intermediate — the paper assumes familiarity with pass@k, token budgets, and outcome-RL training, but its argument is presented through a simple three-part decomposition. Scope (1 sentence): The paper splits the pass@1 gain of 20 released outcome-RL models (1.5B to 32B) over their thinking bases on physics and competition math into finishing, reliability and reach, and adds a controlled 8B-model probe that supplies physics setup knowledge during training.

What This Paper Is About

Outcome RL, which rewards only a correct final answer, reliably raises pass@1, but every pass@1 is measured under a token budget, so one number mixes three different things: answering before the budget runs out, being right once finished, and solving problems the base model cannot solve at all. The paper asks whether a pass@1 gain reflects changed behaviour or new knowledge by decomposing it into finishing, reliability and reach, and then tests whether supplying missing knowledge during training can install capabilities the base does not use on its own.

Key Contributions

1. A budget-aware exact split of the pass@1 gain into finishing (F), reliability (P) and reach, using Eqs. (1)–(5). Unlike pass@k, it separates answering in time from answering right, and it nets reach against the reverse direction (problems the base solves that the RL model does not).
2. A read of twenty released RL–base pairs on thinking bases at each recipe's own training cap and at the next budget above it, the case the authors say Yue et al. (2025) leave open.
3. A comparison rule: compare a recipe with its base at or above its training cap, because below the cap the comparison partly measures how long each model writes.
4. A controlled probe in which an 8B thinking model is given the physical setup of hard problems in five ways during outcome RL, with transfer tested on text-only "twins" of those problems.

Main Findings

• Finishing dominates at tight budgets. At 4k and 8k, finishing accounts for 82% [77, 86] of the change across both sets (71% on physics, 92% on MATH-500). It is the larger part of the change in every recipe up to 7B except Open-RS3, whose score moves by at most 0.004.
• More tokens let the base close most of the gap itself. On physics at 4k and 8k, only 12–65% and 56–90% of base derivations finish; at 16k this rises to 82–93% (physics) and 92–97% (MATH-500). Scored at 64k, the base goes from 0.44 to 0.80 on problems picked at 4k and from 0.38 to 0.59 on problems picked at 8k.
• RL tunes derivation length to the cap. The change in derivation length tracks the change in finishing with Spearman ρ = −0.93. At short caps RL shortens derivations; at long caps it lengthens them and finishes less often, as Polaris-1.7B does even at its own cap. Open-Reasoner-Zero-7B's finishing at 2k falls from 0.98 to 0.65.
• Where the base already finishes, reliability rises. The sixteen recipes that are not cap-bound train at caps where their base finishes 93–100% of derivations. Reliability rises in all sixteen, is the larger part by size in fourteen, and matched reliability (problems both models finish at least seven of eight samples) rises in all sixteen, clear of zero in twelve. At the next budget above the cap, reliability rises in 18 of 20 recipes versus 11 of 20 for finishing.
• The step behind an RL win is usually already in the base. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. Of 26 steps judged decisive where the base was right in one to four of eight samples at 16k, none is missing from the base; the reverse check also finds no base step missing from the RL samples.
• Reach is small. Reach adds at most 2.1 percentage points on any of the four test sets (harder 671-problem physics: −0.006 to 0.018; competition math: −0.002 to 0.021; MATH-500: −0.003 to 0.005). Mean reach across all twenty recipes is +0.003.
• Most lone wins are not new knowledge. Of 144 physics problems only the RL model solved, 3.5% hold up when resampled or read. On four bases at 16k, the base solves 87 of the RL model's 121 lone wins given 56 more samples; 22 differ only in answer form and 9 are real physics. Reading 23 lone wins at the caps finds 2 more real cases.
• Recipes designed to add knowledge reach only slightly further. QuestA, KnowRL and ProRL v2 each robustly solve at most three problems their base gets wrong in all eight. On the harder physics set QuestA and KnowRL lead the twenty recipes in net reach, and 13 of their 19 robust wins stand.
• Larger bases split the same way, with somewhat more reach. For four of the five recipes from 7B up, net reach at the cap is higher than for any smaller recipe (+0.009 to +0.013). The 32B recipe gains mostly reliability (+0.052 gain, +0.047 reliability), and six of its nine robust physics wins are answer form; a more lenient answer judge removes most of its net reach.
• Supplied setup changes training but is not shown to transfer. In the 8B probe every channel changes the model during training (e.g., hint setup claims 55 → 92%; teacher derivation touches 12 → 59% of problems), but none of the four evaluable channels beats plain RL on 87 held-out hard problems. Plain RL's own held-out score rises 0.16 → 0.26 through finishing, with reliability flat.
• Reward works where it lands. Revisiting 60 hard problems raises their solve rate from 0.10 to 0.28 but moves neither the other 293 hard problems nor the held-out set. The setup-reward channel taught the policy to game its scorer, so its held-out effect could not be measured.

Methodology in Plain English

The authors take twenty publicly released outcome-RL models and the thinking models they were trained from, and sample each model eight times per problem at temperature 0.6 with one prompt and one answer checker. They read each pair at 4k, 8k and 16k tokens, at the recipe's own training cap, and at the next budget above the cap on a grid up to 64k. Because pass@k under a budget factors into "did anything finish" times "was a finished sample right," they can split any score change exactly into a finishing term and a reliability term, then add a reach term that counts problems one model solves and the other gets wrong in all eight samples, minus the mirror image to control for chance. Reach is computed at k = 8 with a stricter "robust" version requiring the other model to solve at least k/2 of its samples. Test sets are 300 text-only physics problems, 671 unfiltered physics problems as a check, MATH-500, and 276 recent competition problems. They also run a small controlled experiment on Qwen3-VL-8B-Thinking with GSPO at a 12k-token cap, giving hard physics problems their setup through five channels and testing transfer to easier text-only twins. Blind language-model readers and an author check whether steps in winning derivations are genuinely new.

Why This Matters

The paper changes how reported RL gains should be read: a pass@1 improvement at a fixed budget is mostly a model learning to answer in time or to be right more often on what it already knows, not learning new physics or mathematics. It argues that recipes should be compared with their bases at or above the training cap, and that reach, not pass@1, is the number that speaks to new capability.

Real-world applications:

• Benchmark reporting: budget-matched evaluation prevents length or cap differences from being reported as reasoning improvements.
• Deployment planning: if gains are finishing, teams can often recover much of them by raising the inference token budget rather than retraining.
• Training-data investment: since reach is small, effort may be better spent on coverage and data than on longer RL runs.
• Scientific and technical assistants: physics-style problems, where bases are right on only 21–56% of finished derivations versus 85–91% on MATH-500, need modelling skill that outcome RL is not shown to add.

Industry relevance is direct: teams choosing between more RL training and larger inference budgets, or marketing "new reasoning" from pass@1 gains, now have a decomposition and a comparison rule to check the claim.

Future Directions

• Whether real reach grows with model scale is left open; the 32B recipe shows more apparent reach, but much of it is answer form.
• Larger controlled probes could detect transfer gains smaller than a third of plain RL's effect, and the setup-reward channel needs a scorer that cannot be gamed.
• Why some recipes gain more reliability than others is unexplained, because recipes differ in data, steps and algorithm at once.
• The definition of reach counts problems the base never gets right but leaves some samples unfinished as behaviour; those carry about a tenth to a third of the gain, and how to classify them remains a design choice.

Target Audience

Researchers and engineers working on RL for reasoning models, evaluation and benchmark designers, and anyone interpreting pass@1 or pass@k claims for math and science models. It is most useful to readers who already know what outcome RL and token budgets are and want a precise, budget-aware way to ask what training actually bought.

Authors’ abstract

Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one. On physics and math problems at fixed token budgets, we compare the pass@1 of 20 released RL models (1.5B to 32B) with that of the thinking models they started from. We ask whether each pass@1 gain is changed behaviour or new knowledge. We divide each change in pass@1 into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures problems RL solves that its base answered wrongly in all eight tries, minus problems the base solves that RL answered wrongly in all eight. We find four things. (1) Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing (82% at 4k and 8k). Given more tokens, the base closes most of the finishing gap itself. (2) Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish. (3) By our definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of our four physics and math sets. We checked 144 physics problems that only the RL model solved. Of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. (4) In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.