The Pulse
Study Finds PPO Critics Flatten Value Changes in LLM Reasoning
A preprint identifies Value Flattening in PPO critics and proposes sparse supervision as a way to preserve changes in value across intermediate reasoning states.
AI.info Team ·
A preprint submitted to arXiv on September 16, 2026, reports a failure mode in the value models that help Proximal Policy Optimization train reasoning language models. The authors call the problem “Value Flattening”: the real probability of success can change sharply as a model works through a response, while the PPO critic predicts a comparatively smooth value profile.
“We uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat.”
Yizhuo Li, first-listed author
The paper, titled Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening, lists authors affiliated with Shanghai Jiao Tong University, Shanghai AI Laboratory, Westlake University, Nanjing University, Tsinghua University, the Chinese University of Hong Kong and Nanyang Technological University. Yizhuo Li and Jianhao Yan are marked as equal contributors. Yun Luo, Peilin Zhao, Yafu Li and Yu Cheng are listed as corresponding authors.
The researchers argue that the problem matters because PPO uses the critic’s estimate at each generated token to calculate advantages for policy updates. If the critic misses meaningful changes between intermediate states, the actor can receive weak or distorted signals about which parts of a reasoning trace improved the chance of a correct answer.
The critic sees one outcome repeated across a response
Standard PPO training in the experiments uses terminal-only rewards. A response receives a final binary outcome, and the same sampled return is used as the critic target at every state along that response. The paper distinguishes that repeated training target from the policy-conditioned value: the expected outcome if the current policy continues from a particular intermediate state.
To measure the difference, the authors sample multiple independent continuations from selected intermediate states. Averaging those outcomes produces a Monte Carlo estimate of each state’s value. Across Qwen3 reasoning runs, those estimates often move sharply between adjacent states, while the PPO critic remains relatively flat. In some examples, the critic changes in the opposite direction from the Monte Carlo estimate.
The authors also test a stochastic FrozenLake environment, where a state’s value is the probability of eventually reaching the goal. Holding the training configuration fixed while increasing the maze size makes the critic’s predictions progressively smoother and less accurate, according to the paper. The result suggests that the problem is not limited to language-model responses and may become more pronounced as the state space grows.
Two forces push predictions toward the middle
The paper identifies two mechanisms that can produce the flattening effect. The first comes from the critic’s mean-squared-error objective. When every position in a response receives the same terminal target, the loss can be decomposed into a term that fits the average prediction to the outcome and a second term that directly penalizes variation among the predictions within that response.
Dense token-level supervision therefore discourages the critic from assigning sharply different values to neighboring states, even when the policy-conditioned value changes between them. The effect is especially clear when rewards arrive only at the end of a long response.
The second mechanism comes from temporal correlation. Neighboring language-model states differ by only one newly generated token and share most of their preceding context. The paper’s representation and gradient analysis finds that nearby states produce similar critic updates, so supervising every token can repeat many closely related updates instead of supplying independent information.
SP³O keeps only a few critic targets
To address both effects, the researchers introduce SParse Proximal Policy Optimization, or SP³O. The method leaves the actor objective, rollout procedure and return targets unchanged. It changes only where the critic loss is applied, selecting a small number of well-separated states in each response.
The default setup supervises states at 30%, 60% and 90% of the response, with an additional 95% state for responses at least 6,144 tokens long. The final anchor gives the critic explicit coverage near the end of long responses, where the paper reports a measurable effect on performance and repetition.
On Qwen3-4B-Base, SP³O reduced critic profile mean-squared error against policy-conditioned Monte Carlo values by 36% at 30% response progress, 11% at 60% and 21% at 90%. The paper also reports that the median effective rank of response-centered critic representations increased from 4.33 with PPO to 5.63 with SP³O, a result the authors interpret as evidence that the sparse method preserves more information about differences between states.
Three anchors beat dense supervision in the reported tests
The authors evaluate Qwen3-4B-Base and Qwen3-8B-Base trained on DAPO-Math-17k. Their evaluation includes seven mathematical-reasoning tasks, including AIME24, AIME25, AIME26, AMC23, MATH500, Minerva and OlympiadBench, plus six out-of-distribution benchmarks: ARC-C, MMLU-Pro, GPQA, AGIEval-English, BigBenchHard and ZebraLogic-Grid.
For Qwen3-4B-Base, standard PPO records a 37.60% average on the mathematical suite, while SP³O reaches 45.57%. On the out-of-distribution suite, the corresponding averages are 51.95% and 59.28%. For Qwen3-8B-Base, SP³O reaches 50.51% on the mathematical suite and 66.37% on the out-of-distribution suite, compared with 48.50% and 64.38% for standard PPO.
Those results come from the authors’ reported 24K evaluation setting. Mathematical scores average 32 generations, while the out-of-distribution scores average four generations. The paper also compares SP³O with GRPO, a critic-free method, and reports higher mathematical and out-of-distribution averages for SP³O in both model-size blocks.
More supervision does not improve the critic
An ablation on Qwen3-4B-Base finds that three, four and eight supervised states per response produce higher mean training rewards than denser configurations. Performance peaks at three anchors in the reported experiment, then drops substantially at 16 and 64 supervised states, moving closer to the dense PPO baseline.
Anchor placement also matters. Three fixed positions at 20%, 50% and 80% produce a 44.65% mathematical average, while the paper’s 30%, 60% and 90% placement reaches 45.57%. Random placement scores 36.59%, below the 37.60% PPO baseline. Adding the late-tail anchor raises the reported result from 44.10% to 45.57% and reduces a repetition measure from 18.33 to 1.12.
The study is a preprint, and its evidence comes from the authors’ experiments rather than an independent replication. Its narrower claim is specific: under the terminal-reward, token-level critic training setup examined here, PPO can preserve response-level discrimination while losing resolution between states inside the same response. SP³O’s reported gains come from applying the critic loss at three widely separated positions instead of every token.
Read the full preprint on arXiv.