Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning

AI summary

Overview

Research area: Natural Language Processing, specifically reinforcement learning for improving the reasoning ability of large language models (RLVR — Reinforcement Learning with Verifiable Rewards), with a cognitive-science motivation drawn from the human thinking theory of Path Dependence.

Technical level: Advanced. The paper builds on GRPO/DAPO-style policy-gradient objectives, importance sampling ratios, advantage normalization, and token-level gradient masking, and it assumes familiarity with RLVR training pipelines.

One-sentence scope: The paper identifies a "Beginning Lock-in Effect" in LLM reasoning — early tokens strongly constrain the whole reasoning trajectory — and exploits it with a training method that optimizes only the prefix portion of generated outputs, reporting accuracy gains on math, physics, chemistry, and biology reasoning benchmarks.

What This Paper Is About

Standard RLVR methods train a language model on every generated token, treating all tokens as equally worth optimizing. The authors argue this wastes effort on low-impact tokens and limits how much the truly important tokens can improve. Their goal is to show that the beginning of an answer matters disproportionately, and to build a training algorithm (PPPO) that concentrates optimization on that beginning.

Key Contributions

1. Identification of the Beginning Lock-in Effect (BLE). Through controlled experiments on AIME'24 and GPQA Diamond with DeepSeek-R1-Distill-Qwen models, the authors show that reasoning from prefix tokens taken from incorrect outputs drives accuracy down (up to 27.5% on AIME'24 with DeepSeek-R1-Distill-Qwen-14B), while prefix tokens from correct outputs improve accuracy (up to 20.2%).

2. Progressive Prefix-token Policy Optimization (PPPO). A new RLVR method that retains gradients only for the first η fraction of generated tokens and masks the gradients of all subsequent tokens, so updates are applied exclusively to the prefix reasoning segment.

3. Two supporting training strategies. (a) Progressive Prefix Retention, which starts with short prefix windows and lengthens them as training plateaus; (b) Continuation Accumulated Reward, which samples multiple continuations from one fixed prefix and accumulates their scores as the reward signal for that prefix.

4. Broad empirical validation. Experiments across three Qwen3 backbone sizes and five reasoning benchmarks show PPPO outperforming the representative RLVR baselines GRPO, DAPO, INTUITOR, and DAPO-FT.

Main Findings

• BLE is real and bounded to the early tokens: Model performance shows a significant transition when the retention proportion reaches 15% and stabilizes at 35%, which the authors interpret as BLE manifesting within the first 15% of tokens and essentially establishing by 35%. Accordingly, PPPO initializes η at 15% and increases it to 35% in steps of 5%.

• LLMs struggle to recover from bad starts: Even after inserting "wait" and "however" after malignant prefix tokens to encourage reflection, the maximum recovery of accuracy reaches only 9.2%.

• PPPO wins almost everywhere: PPPO achieves the highest accuracy in 14 out of 15 settings (3 backbone models × 5 benchmarks). The single exception is on AMC'23 with Qwen3-1.7B, where PPPO slightly underperforms DAPO by 0.14%.

• Size of gains: Compared with representative RLVR methods, PPPO increases accuracy by up to 18.02% on a single benchmark (AIME'25) and increases average accuracy across all benchmarks by up to 14.64%. The abstract summarizes this as accuracy improvements of 18.02% on only 26.17% training tokens.

• Strongest absolute numbers on the largest backbone: With Qwen3-8B, PPPO reaches 72.19 on AIME'24, 59.69 on AIME'25, 94.73 on MATH 500, 86.75 on AMC'23, and 58.13 on GPQA Diamond, for an average of 74.30.

• Better training efficiency: Using the defined metric LE (ratio of Average Accuracy Increase, AAI, to Proportion of Optimized Tokens, POT), PPPO achieves the highest LE among all baselines — up to 14.64% accuracy improvement while optimizing only 24.83% of generated tokens (Qwen3-8B).

• Cheaper reasoning at inference: PPPO produces the lowest reasoning cost among the compared methods, reducing the average number of generated tokens by up to 18.35%.

• Prefixes trained by PPPO transfer: Requiring Qwen2.5-7B-Instruct to reason from prefix tokens extracted from Qwen3-4B, prefixes from the PPPO-trained model yield optimal performance across all benchmarks, improving accuracy by up to 9.04%, whereas prefixes from untrained Qwen3-4B give only marginal gains and even reduce accuracy on AIME'25.

• Accumulated reward stabilizes training: With Qwen3-4B and 4 training runs, a single-sample reward (G = 1) gives avg@4 of 60.46 with variance 3.30; using multiple continuations raises this to 66.11 (G = 4, var 1.47), 69.36 (G = 8, var 0.63), and 69.53 (G = 16, var 0.56). The paper reports average accuracy increases of up to 8.37% and variance reductions of up to 2.67%.

• Progressive retention helps: Compared with 5 alternative prefix retention strategies, the progressive scheme stabilizes learning and increases average accuracy by up to 12.49%; relative to a fixed "η = 35" strategy it increases average accuracy by 2.19% while reducing training tokens by 8.83%.

Methodology in Plain English

The authors first ran a diagnostic study. For each problem in AIME'24 and GPQA Diamond, they used DeepSeek-R1-Distill-Qwen models to generate 4 correct and 4 incorrect answers, cut each answer off after its first 15% of tokens, and then forced the original model to continue from those fixed prefixes eight times per prefix. Continuing from prefixes drawn from wrong answers consistently hurt accuracy; continuing from prefixes drawn from right answers consistently helped. They then tested whether the model could be rescued from a bad start by appending "wait" or "however" — it largely could not.

Building on that, PPPO changes what part of an output gets trained. For each question the policy model samples N outputs; the first η fraction of tokens of each output becomes a fixed prefix. From each prefix, the model generates G continuations, and the reward for that prefix is the accumulated count of correct answers among those continuations plus the correctness of the original full output. Advantages are computed by normalizing rewards within the group, exactly in the style of GRPO, and only the tokens inside the prefix receive gradient updates — the rest are masked out via an indicator function. During training, η is raised only when validation accuracy stops improving, creating an easy-to-hard curriculum. Experiments use Qwen3-1.7B, Qwen3-4B and Qwen3-8B in thinking mode, trained on DAPO-Math-17K with a learning rate of 1×10⁻⁶, 8 sampled outputs, 8 continuations, ε_low = 0.2 and ε_high = 0.28, a maximum response length of 10240 tokens, and 8×8 sampled outputs per step versus 64 for all baselines. Evaluation is zero-shot, run 32 times per dataset, reported as avg@32 on AIME'24, AIME'25, MATH 500, AMC'23, and GPQA Diamond.

Why This Matters

Impact on research. The paper reframes RLVR training efficiency as a question of which tokens to optimize rather than how many. It connects a cognitive-science idea (Path Dependence, David 1975) to an observable model behavior, and suggests that token-level heterogeneity in RLVR — previously addressed by targeting high-entropy tokens such as "wait," "however," and "rethink" — may be better handled by targeting the prefix. Because PPPO is a modification to the optimization objective rather than a new reward model, it is conceptually portable to other RLVR pipelines.

Real-world applications (as directions this technique could support):

• Mathematical and scientific problem solving assistants that need to produce correct multi-step reasoning, the exact setting of the AIME'24, AIME'25, MATH 500, AMC'23 and GPQA Diamond benchmarks used here.
• Cost-sensitive deployment of reasoning models, where the reported reduction of up to 18.35% in generated tokens lowers inference cost while maintaining accuracy.
• Distillation or guiding setups, where high-quality prefix tokens from a trained model improved a different model (Qwen2.5-7B-Instruct) by up to 9.04%.
• Training-data efficiency, where accuracy gains are reported while optimizing as little as 24.83% of generated tokens.

Industry relevance. The two levers the paper targets — training tokens (POT down to 24.83% in the best reported setting) and inference tokens (up to 18.35% fewer) — map directly onto GPU training cost and serving cost. The LE metric (AAI/POT) is also a directly comparable efficiency figure that practitioners can track against GRPO, DAPO, INTUITOR and DAPO-FT baselines. The paper does not report wall-clock training time or hardware used, so cost conclusions should be drawn from the token counts it does report.

Future Directions

• Determine what happens beyond 35%: The paper shows η stabilizes at 35% and caps its schedule there; whether raising η further helps, plateaus, or eventually harms training is left open.
• Test the transfer of PPPO-trained prefixes more broadly: Only one cross-model experiment is reported (Qwen2.5-7B-Instruct reasoning from Qwen3-4B prefixes); the generality of prefix transfer across model families and scales is unresolved.
• Apply the method outside verifiable-reward domains: All benchmarks here are math and science tasks with checkable answers; whether prefix-focused optimization helps open-ended reasoning or tasks without automatic verifiers is not addressed.
• Reconcile prefix optimization with high-entropy-token methods: The authors argue that methods focusing on high-entropy tokens (DAPO-FT) give unstable improvements, but a systematic comparison of what prefix tokens versus high-entropy tokens actually encode is not provided.

Target Audience

Researchers and engineers working on LLM post-training and reinforcement learning, particularly those using GRPO-, DAPO-, or PPO-style RLVR pipelines who care about training efficiency and reasoning quality. It is also relevant to cognitive-science-informed AI researchers interested in how human thinking theories such as Path Dependence can generate testable hypotheses about LLM behavior. Readers need comfort with policy-gradient objectives and token-level loss masking to follow the methodology sections; the diagnostic findings and benchmark tables are accessible to a broader audience.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capability of Large Language Models (LLMs). Current RLVR approaches typically conduct training across all generated tokens, but neglect to explore which tokens (e.g., prefix tokens) actually contribute to reasoning. This uniform training strategy spends substantial effort on optimizing low-return tokens, which in turn impedes the potential improvement from high-return tokens and reduces overall training effectiveness. To address this issue, we propose a novel RLVR approach called Progressive Prefix-token Policy Optimization (PPPO), which highlights the significance of the prefix segment of generated outputs. Specifically, inspired by the well-established human thinking theory of Path Dependence, where early-stage thoughts substantially constrain subsequent thinking trajectory, we identify an analogous phenomenon in LLM reasoning termed Beginning Lock-in Effect (BLE). PPPO leverages this finding by focusing its optimization objective on the prefix reasoning process of LLMs. This targeted optimization strategy can positively influence subsequent reasoning processes, and ultimately improve final results. To improve the learning effectiveness of LLMs on how to start reasoning with high quality, PPPO introduces two training strategies: (a) Progressive Prefix Retention, which shapes a progressive learning process by increasing the proportion of retained prefix tokens during training; (b) Continuation Accumulated Reward, which mitigates reward bias by sampling multiple continuations for one prefix token sequence, and accumulating their scores as the reward signal. Extensive experimental results on various reasoning tasks demonstrate that our proposed PPPO outperforms representative RLVR methods, with the accuracy improvements of 18.02% on only 26.17% training tokens.