RLVR Sharpens the Pencil, Not the Mind

AI summary

Overview

Research area: Machine learning interpretability and post-training — specifically, what Reinforcement Learning with Verifiable Rewards (RLVR) actually changes inside a language model, not just in its outputs.

Technical level: Intermediate. The paper is written clearly, but readers will get the most from it with some familiarity with pass@k evaluation, linear probes, and AUROC as a detection metric.

Scope: A single-model (OLMo-2-1B), single-task (GSM8K grade-school arithmetic) study tracking all 13 released RLVR checkpoints to test whether internal representations of reasoning errors change as behavior improves.

What This Paper Is About

RLVR trains models by rewarding correct final answers without supervising the reasoning process, and models trained this way appear to become much better reasoners. The leading explanation, "sharpening," holds that the base model already contains the needed reasoning and RLVR simply makes it more likely to use its best solution paths. Existing evidence for this comes either from comparing outputs before and after training, or from comparing a base model to a trained model at a single endpoint — nobody has watched internal representations evolve across the training trajectory. This paper does exactly that, measuring both behavior and internal error-detection signals at every checkpoint to ask whether the model's internal picture of a reasoning mistake sharpens as its accuracy rises.

Key Contributions

1. A trajectory-level test of sharpening. Rather than comparing a base model to a final model, the authors probe all 13 RLVR checkpoints of OLMo-2-1B (steps 200–2600, interval 200), giving checkpoint-by-checkpoint resolution on both outputs and internal states.

2. Evidence that coverage does not narrow in this setting. pass@1 rises from 0.580 to 0.613 while pass@256 rises from 0.945 to 0.969 — the opposite of the boundary-narrowing reported for larger models — which removes mode collapse as a confound and makes the representational question clean.

3. A controlled error-injection probing pipeline. Three types of arithmetic error (a +3 operand perturbation, an operator swap, and a result-scale error) are planted into correctly solved problems, with logistic-regression probes trained on PCA-reduced hidden states at six readout sites to detect them at each checkpoint.

4. A positive control that validates the null. The same probing pipeline detects real training-induced change at the final answer token (AUROC 0.93) while landing near chance on intermediate reasoning steps, showing the flat trajectory on error representations is a meaningful null rather than a weak method.

Main Findings

• Accuracy improves modestly, coverage grows: First-try accuracy goes from 0.580 to 0.613. An apparent pass@32 drop is described as an artifact of small k; pass@256 rises from 0.945 to 0.969, meaning the set of problems the model can eventually solve grows rather than shrinks.

• Error representations stay flat: For the +3 operand error, error-detection AUROC spans 0.798–0.840 across all 13 checkpoints, a spread of 0.042 — smaller than the CI half-width of approximately 0.044. Operator-swap shows a spread of 0.032 (R² = 0.005) and result-scale a spread of 0.033 (R² = 0.345); drift is statistically indistinguishable from zero for all three.

• The signal is computed, not a surface cue: A norm proxy sits at chance throughout (0.50–0.52). Probe AUROC rises from 0.498 at layer 0 to a mid-layer peak of 0.833, with near-identical layer profiles across all 13 checkpoints — so the error information is a mid-layer computation, not token-length or activation-magnitude shorthand.

• Change is localized at the output token: A probe distinguishing early from late checkpoints reaches AUROC 0.93 at the output token for both operator-swap and result-scale, with the layer-0 baseline at chance, while the same probe on intermediate reasoning tokens lands near chance. Change exists, and it is not where the arithmetic reasoning happens.

• Robustness across analysis choices: Probing all 17 layers, the cross-checkpoint spread stays within noise at 16 of 17 layers; the one marginal exception is layer 13 (spread 0.073, CI half-width 0.031, mean AUROC 0.864, in a late layer where signal is already declining). Probing all six readout sites, every site is invariant (spread ≤ 0.034).

• Layer-sweep details: In the five-checkpoint layer sweep, layer-0 AUROC is 0.500–0.540 (mean 0.517), profiles peak around layer 6 (mean 0.983) with a spread at the peak of only 0.008 against a CI half-width of 0.009, then decline. Absolute AUROC is roughly 0.10 higher than in the main analysis because the ablation used a larger chain sample (n_repair ≈ 208 vs. ≈ 174), but the spread — the actual invariance measure — is tighter (mean 0.022 vs. 0.042).

• Readout-site details: At layer 11, spreads range from 0.000 (L_inj_step, mean AUROC 1.000) to 0.034 (L_dep_num, mean 0.933); decodability is highest near the injection site and attenuates downstream, and that gradient shape is itself preserved across checkpoints.

• Protocol does not drive the trend: Across a 2 × 2 grid of decoding (greedy vs. sampled at temperature 0.8) and parser (strict vs. lenient last-number) on a 200-problem subset, parser choice matters by at most 0.002, greedy runs roughly 3 points above sampled (mean 0.646 vs. 0.615), and the +3.3-point upward trend appears under both decoding schemes.

• Practical consequence: Probes or verifiers trained to detect reasoning errors in the base model should transfer across RLVR checkpoints, amortizing verification compute.

Methodology in Plain English

The authors download OLMo-2-1B-Instruct at every one of its 13 published RLVR checkpoints (steps 200 through 2600) and run two procedures at each.

Behavioral sweep: They sample many completions per GSM8K test problem (1319 problems total) at temperature 0.8 and compute pass@k, using a wide sweep (K = 32, n = 200 problems, all 13 checkpoints) and a deeper sweep (K = 256, n = 128 problems, a 5-checkpoint subset) to measure how much solution coverage exists.

Representational probe: They take problems the model already solves, deliberately plant one of three arithmetic mistakes into an early reasoning step, and regenerate the rest of the chain. Each chain is labeled "repair" if the final answer is still correct or "carry" if the answer flips. They then extract hidden states at six readout sites along the chain (from the corrupted number to the final answer token), fit PCA with 256 components on the combined clean and corrupted activations, and train logistic-regression probes with 5-fold cross-validated AUROC and bootstrap confidence intervals. The pre-registered reporting site is layer 11 at the token of the step depending on the corrupted step.

The key logic is comparative: if RLVR sharpens internal error representations, probe AUROC should climb across checkpoints. It does not. To prove the pipeline could detect change if it existed, they run the identical probe to distinguish early-checkpoint from late-checkpoint activations — and it succeeds dramatically at the answer token while failing at the reasoning steps.

Why This Matters

Impact on research: The sharpening account has rested on output-level evidence plus single-endpoint mechanistic comparisons. This paper fills the trajectory-level gap and localizes where RLVR's representational footprint actually lives — the output distribution rather than the arithmetic-error substrate. It also situates its result within recent work predicting that RLVR-induced distributional shifts are sparse, placing arithmetic error representations in the "invariant majority." Its methodological template (a positive control that validates a null) is reusable for other interpretability nulls.

Real-world applications:

• Reusable verifiers: A reasoning-error detector trained once on a base model could be deployed across RLVR checkpoints without retraining, cutting verification compute in inference pipelines.
• Cheaper safety monitoring: If internal error signals are stable through post-training, monitors built at one point in a model's lifecycle need less re-validation after each training run.
• RLVR evaluation design: Teams can separate "did behavior change" from "did the model's internal reasoning change," avoiding over-attribution of capability gains to representational learning.
• Checkpoint selection: Because pass@1 and pass@256 move differently, practitioners get a clearer picture of whether a training run is adding capability or just reweighting existing capability.

Industry relevance: Post-training pipelines are expensive, and knowing that RLVR acts on output selection rather than on the reasoning substrate informs where to invest — in data and reward design that surfaces existing solutions versus in methods that might genuinely install new reasoning.

Future Directions

• Scale and diversity: Replicating the trajectory-level probe across larger models and non-arithmetic tasks (where ground-truth reasoning errors are harder to define unambiguously) to see whether the invariance holds or whether arithmetic is a special case.
• Causal, not just correlational, tests: The authors acknowledge linear probes show information is present, not that the model uses it; intervention or steering experiments across checkpoints would test whether the stable error representation is causally implicated in outputs.
• Reconciling with boundary narrowing: Since pass@256 grows here but narrows in the larger models studied by others, pinning down what controls that difference — model scale, task, or RLVR setup — is an open question.
• Transfer of verifiers in practice: Measuring whether a probe trained on one checkpoint really does maintain detection performance on held-out checkpoints and other task distributions, which the paper suggests but does not test end to end.

Target Audience

Interpretability researchers and post-training practitioners who want to know what RLVR does beneath the output distribution. It is especially useful for engineers building verifiers or safety monitors on top of checkpoints, and for readers evaluating claims that RL models "learn to reason" — the paper's disciplined single-model, single-task scoping (the authors explicitly limit claims to one model and arithmetic) makes it a good model of careful negative-result reporting. The paper does not report training-side details of the RLVR procedure itself, so readers seeking recipe-level guidance will need to consult its cited sources.

Authors’ abstract

Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening---that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.