TPPO: Tracking-Value PPO For Language Model Reasoning

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for large language model reasoning, specifically value function learning and token-level credit assignment inside Proximal Policy Optimization (PPO).

Technical level: Intermediate to Advanced. The paper assumes familiarity with Markov decision processes, PPO, Generalized Advantage Estimation (GAE), and advantage functions, and it derives a Bayesian decomposition of prefix values in log-odds space. The core idea is explainable without the derivation, but the justification is mathematical.

Scope: The paper introduces a new critic parameterization for PPO — Tracking-Value PPO (TPPO) — that models how the estimated probability of success changes token by token, and evaluates it on mathematical reasoning, Countdown, and scientific coding tasks across several Qwen model sizes.

What This Paper Is About

In RLVR, a language model receives a single reward at the end of a response (correct or incorrect), so every token in a response shares the same target. PPO trains a critic to estimate the probability of eventual success from each prefix, and the difference between successive prefix values determines which tokens get more or less credit. The paper shows that standard PPO critics often fail to predict even the direction of these changes — in one example, the Monte Carlo estimate of the value change was −0.992 across an incorrect claim, while the PPO critic predicted +0.036, reversing the relative credit ordering — and proposes a critic that explicitly accumulates token-level increments so value evolution is tracked rather than estimated independently at each prefix.

Key Contributions

1. Characterizing value evolution along reasoning. The authors measure a discrepancy between Monte Carlo estimates of how prefix values change and the changes captured by standard PPO critics, showing that sign accuracy stays near chance and even declines during training (from 50.34% to 45.91%), which highlights the difficulty of learning fine-grained credit signals under sparse terminal rewards.

2. A cumulative parameterization of prefix values. They derive that successive prefix values admit a Bayesian update that is additive in log-odds space (Eq. 8), with each token contributing a log-likelihood ratio term. TPPO exploits this by summing position-wise scalar critic outputs into value logits and applying a sigmoid (Eq. 9), trained only with terminal-outcome supervision on the reconstructed values.

3. Improved policy optimization without extra supervision. The reconstructed values plug directly into standard GAE and the standard PPO clipped objective. TPPO requires no additional rollouts and no process-level annotations, yet improves reasoning performance and the quality of training signals across models and tasks.

4. Diagnostic evidence for why it works. Controlled comparisons isolate the parameterization from the loss, and checkpoint-level analyses show better tracking of value evolution and advantage estimates that agree more closely with a Monte Carlo reference.

Main Findings

• Aggregate math gains over PPO (n = 8): TPPO improves the in-domain average by 7.0 points on Qwen3-4B-Base (41.6 vs. PPO's 34.6) and 8.4 points on Qwen3-8B-Base (45.9 vs. PPO's 37.5), also outperforming GRPO, VAPO, and SPPO on those averages.

• Out-of-distribution generalization improves: On the OOD average, TPPO reaches 66.9 versus PPO's 62.8 on Qwen3-4B-Base, and 69.7 versus PPO's 67.6 on Qwen3-8B-Base (n = 8).

• Gains persist at n = 1: TPPO improves over PPO by +7.5 points on Qwen3-8B-Base (44.9 vs. 37.4) and +10.1 points on Qwen3-4B (64.1 vs. 54.0).

• Directional accuracy of value evolution: In a controlled comparison with a fixed actor on Qwen3-8B-Base, the TPPO critic with BCE improves sign accuracy over a standard critic with BCE by 2.01 percentage points at τ = 0 (52.71 vs. 50.70) and 2.80 percentage points at τ = 1 (53.94 vs. 51.14). The standard MSE critic scored 50.51 at τ = 0 and 51.19 at τ = 1.

• Tracking is maintained during training (RQ2): Across the last four checkpoints, TPPO's advantage in directional agreement with Monte Carlo estimates is 3.9–6.8 percentage points for τ = 0 and 4.9–7.4 for τ = 1, with 95% bootstrap intervals for the gains above zero, while PPO's agreement declines.

• Better credit assignment (RQ3): Measured against GAE advantages computed by substituting Monte Carlo estimates for critic values (λ = 1), TPPO's advantage sign accuracy improves by 15.3–19.9 percentage points over the last four checkpoints, with 95% bootstrap intervals above zero.

• Better value estimates: On AIME responses, TPPO shows lower mean absolute error and higher value accuracy at every evaluated reasoning position than PPO; its error grows toward the end of a response but stays below PPO's.

• Both design choices matter: An ablation shows PPO with BCE improves over PPO but is still below TPPO, while TPPO with MSE deteriorates early and stays well below the other variants. The paper attributes the main gains to the cumulative parameterization, with BCE as an additional benefit.

• Robust to the GAE λ coefficient: When varying only the actor's GAE λ over {1, 0.95, 0} (with the critic's λ fixed at 1), TPPO retains its advantage over PPO in every evaluated setting.

• Gains extend to other tasks and models: On Countdown with Qwen2.5-Math-1.5B, TPPO improves by +4.6 on 3–4 input numbers and +3.2 on 5 input numbers. On scientific coding with Qwen3.5-9B, average accuracy improves by +7.0, including +15.5 on LiveCodeBench Code Gen.v6 (73.1 vs. PPO's 57.6), +8.5 on APPS Introductory, +5.4 on HumanEvalFix, +3.0 on HumanEval, and +2.4 on SciCode.

• Training stability: On Qwen3-8B-Base (n = 8), TPPO establishes an early lead in validation accuracy and Pass@4 and keeps it, with smaller actor gradient norms and slower entropy decay than PPO.

Methodology in Plain English

The authors start from a simple observation about what a critic is supposed to compute. Under a binary terminal reward with γ = 1, the value of a prefix is just the probability that the continuation from that prefix ends up being correct. When the model emits one more token, that probability gets updated by a Bayesian rule: the new probability depends on how likely that token is under correct completions versus incorrect ones. Taking the log-odds of both sides turns this update into a plain addition — the log-odds of the new value equals the log-odds of the old value plus one token-specific term. Unrolling this along a whole response means the value of any prefix can be written as a sigmoid applied to a running sum of per-position terms.

TPPO builds the critic around that structure. Instead of predicting an independent value for every prefix, the critic emits one scalar at each position; these are added up into a running logit, and a sigmoid converts the running logit into a prefix value. Every reconstructed value is then trained against the same binary terminal outcome using a binary cross-entropy loss, so the individual increments receive gradients from all subsequent reconstructed values that depend on them. No intermediate labels, reward models, or extra sampling are used.

The resulting values replace the standard critic's outputs inside GAE, and the policy is updated with the usual PPO clipped objective. Because the per-position increments can be positive or negative, the critic can naturally represent a value drop at a bad reasoning step and a value rise at a good one, which a standard independently-parameterized critic does not explicitly model.

The evaluation protocol is standard for this literature but carefully instrumented. Math training uses roughly 30k examples from Polaris with n ∈ {1, 8} sampled responses per prompt, temperature 1.0, and an 8,192-token maximum training length, using the Slime framework on nodes with eight H200 GPUs each. Evaluation spans ten math benchmarks, reporting Avg@32 on AIME and AMC and Avg@4 on the rest at temperature 1.0 with a 24k-token maximum response length. To check whether the critic is actually tracking value changes, the authors evaluate five Qwen3-8B-Base checkpoints on a fixed subset of 64 training prompts, with 8 sampled responses per prompt (roughly 5.1k rollouts in total), estimate Monte Carlo values at nine anchors spanning 10–90% of each response with 128 continuations per anchor (approximately 5.9M valid continuations in total), and measure sign agreement with those estimates.

Why This Matters

Impact on research. The paper reframes critic design for LLM reasoning as a problem of modeling value evolution rather than estimating prefix values in isolation. It provides a principled justification (a Bayesian identity in log-odds space) for a cumulative critic, and it introduces a diagnostic — sign agreement between critic-predicted and Monte Carlo-estimated value changes — that other researchers can reuse. It also shows that better credit assignment is achievable without additional rollouts or process-level supervision, which are the two resources most prior approaches to dense credit have required.

Potential real-world applications (the paper itself evaluates only on benchmarks and does not report deployed applications):

• Training reasoning models for mathematics, competition problems, and quantitative tutoring systems.
• Improving code-generation and code-repair assistants, where the paper reports gains on LiveCodeBench, HumanEval, HumanEvalFix, APPS, and SciCode.
• Constraint-satisfaction and puzzle-style tasks such as Countdown, which resemble planning and scheduling problems.
• Any RLVR pipeline where a single correctness signal must be distributed across a long chain of thought, including scientific and symbolic reasoning.

Industry relevance. Because TPPO changes only the critic parameterization and loss, it can be dropped into existing PPO-based RLVR stacks — the paper uses the open-source Slime framework and standard Math-Verify outcome rewards, and the code is released at https://github.com/LINs-lab/TPPO. The reported gains appear at both n = 1 and n = 8 rollout budgets and across 1.5B to 9B-scale models, which matters for teams whose compute budget limits how many rollouts they can afford. The claim of more stable training (smaller actor gradient norms, slower entropy decay) is also relevant to practitioners dealing with policy collapse in long-chain-of-thought training.

Future Directions

1. Scaling beyond 9B parameters. The paper's own limitations section states that experiments cover models up to 9B parameters and that effectiveness at substantially larger scales is unverified. The main experiments use Qwen3-4B-Base and Qwen3-8B-Base, with Qwen3.5-9B used for coding.

2. Extending beyond binary terminal rewards and γ = 1. The current formulation assumes a binary reward signal and a discount factor of 1; the authors note that extending it to dense or non-binary rewards requires further investigation.

3. Broadening task coverage. The evaluation covers mathematical reasoning, Countdown, and scientific coding. The authors state that broader task coverage is needed to assess generality beyond these verifiable domains.

4. More precise critic diagnostics. Monte Carlo values are estimated at selected anchors on a subset of prompts, which captures changes between anchors but can miss fluctuations within each interval, and finite MC samples introduce estimation noise — a trade-off the authors acknowledge under computational constraints.

Target Audience

Researchers and engineers working on reinforcement learning for language models — particularly those implementing or studying PPO critics, GAE, and credit assignment for long chain-of-thought reasoning. It is also relevant to practitioners training reasoning or coding models with RLVR who want to improve credit assignment without adding process supervision or extra rollout sampling. Readers without a background in RL, MDPs, or advantage estimation will find the method section demanding, though the core idea and the empirical evidence are accessible at a higher level.

Authors’ abstract

Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While \emph{value evolution along trajectory} characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. \method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in https://github.com/LINs-lab/TPPO.