S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) reasoning — specifically reinforcement learning with verifiable rewards (RLVR) applied to long chain-of-thought (CoT) tasks, and the actor–critic training methods used within it.

Technical level: Intermediate. The paper assumes familiarity with policy-gradient methods (PPO, GRPO, GAE) and with the standard RLVR setup of a policy generating a response and a verifier returning a scalar reward, but it explains its own design choices from first principles.

Scope (one sentence): The paper presents S3AC, an actor–critic framework that makes a trained value model practical for long-CoT RLVR by fixing critic pretraining instabilities and by reusing the critic to select which prompts to train on.

What This Paper Is About

Most RLVR systems for LLM reasoning avoid training a critic entirely: methods like GRPO estimate a baseline by sampling a group of responses per prompt, while REINFORCE++ and FlashREINFORCE use batch statistics. The paper argues these critic-free designs lose two things — a baseline specific to each prompt's difficulty, and any per-token credit assignment within a response — and that this matters because a failure on a hard prompt and a failure on an easy prompt carry very different information. Its goal is to show that a critic can be trained reliably for long reasoning trajectories, and that once trained it improves both the learning signal and the choice of which prompts to generate from.

Key Contributions

1. A practical critic-training recipe for long-CoT RLVR. It combines coverage-oriented value pretraining on trajectories from policies of different strengths, a small-scale value-head initialization with standard deviation 1/(H+1), and a relaxed critic gradient-clipping threshold of 10 instead of 1.

2. Token-specific advantages and prompt-specific baselines from one critic. The critic is updated online and used with GAE to produce token-level advantages, with actor and critic GAE decoupled (λ_critic = 1, λ_actor = 1 − 1/(αT) with α = 1.5).

3. Difficulty-Aware Prompt Sampling (DAPS). The critic's initial-state value is converted into a success-rate estimate, blended with observed pass/fail history, and used to accept prompts of intermediate estimated difficulty before generation — avoiding both offline difficulty initialization and a separate value model.

4. Evaluation across four models from 1.5B to 397B parameters, covering group rollout and single rollout, with comparisons against GRPO-X, REINFORCE++, and FlashREINFORCE.

Main Findings

• Group rollout gains over a modern GRPO variant. On Intern-S2-Preview-35B, the best-checkpoint seven-benchmark average rises from 76.4 (GRPO-X) to 80.5 (S3AC), with gains on all seven benchmarks. On Intern-S2-397B, the average rises from 85.1 to 87.3, leading on all five benchmarks (GPQA-D 90.5 vs 88.9, IMO-AnswerBench 84.2 vs 81.4, MMMU-Pro 82.9 vs 80.8, MathVision 89.7 vs 87.5, HMMT26 88.8 vs 86.9).

• Gains also at 1.5B. Qwen2.5-Math-1.5B group rollout averages 43.3 for S3AC versus 42.7 for GRPO-X; R1-Distill-1.5B averages 31.4 versus 29.5.

• Single-rollout stability at 35B. S3AC reaches 80.9 on the 35B seven-benchmark average versus 70.4 for REINFORCE++ — a 10.5-point gap — and 67.4 for FlashREINFORCE. Both critic-free baselines improve early and then decline, while S3AC retains most of its gains through 1,600 gradient steps. REINFORCE++ shows rapid growth in rollout entropy and response length, while FlashREINFORCE shifts toward shorter responses as its score declines.

• Critic values track prompt difficulty. With a frozen 35B actor–critic pair, initial critic values track mean Monte Carlo returns per prompt. In long failed responses (over 20K tokens) that all receive a terminal reward of −1, mean raw advantages stay near zero for hard prompts but below −1 for easy prompts.

• Critic values track within-response progress. On Intern-S2-Preview-35B, the critic value rises when the response identifies candidate values and increases further when it determines the requested term, 4/5; in a numerical example it rises sharply toward zero after the response computes 0.1622 and matches it to option H (0.1614).

• Coverage is a real bottleneck. Among responses longer than 24K tokens, only 5.0% are correct in initial-policy samples versus 61.8% for the policy at gradient step 1,600. A coverage critic (327,680 trajectories, 7.08B response tokens) beats an on-policy critic (65,536 trajectories, 0.74B response tokens) on early explained variance and value loss, reaching 80.78 versus 79.86 at 1,600 actor gradient steps.

• The value head dominates early gradients. The head accounts for over 99% of the squared gradient norm during the first 10 critic updates on the 35B model despite having only 2,048 parameters. Lowering initialization standard deviation to 1/(H+1) cuts early value loss from 0.72 to 0.45. Raising the clipping threshold from 1 to 10 reduces the fraction of clipped critic updates from 99.7% to 12.5%, with comparable final value loss.

• DAPS improves performance in both rollout settings. Within matched gradient-step budgets, best scores improve from 79.04 to 80.49 (35B group), 74.40 to 76.94 (35B single), 30.87 to 31.08 (R1-Distill group), and 30.16 to 30.36 (R1-Distill single). DAPS also reduces trajectories discarded by the group filter, and shifts training share toward prompts with reference success rate in (0.2, 0.8].

• Cost. S3AC requires 27–40% more amortized wall time per actor gradient step than GRPO-X.

Methodology in Plain English

The setup is standard RLVR: the model writes a full reasoning response to a prompt, and a verifier gives a single reward of +1 or −1 at the end. The question the authors tackle is how to compute a useful learning signal from that one number at the end.

Their approach is to train a second model — the critic — that predicts, at every point in the response, how likely the response is to end in success. Early in the response, that prediction depends only on the prompt, so it acts as a difficulty-aware baseline. Later in the response it depends on the reasoning so far, so it can tell whether a particular step moved toward or away from a correct answer.

Training this critic turns out to be the hard part, and they fix two specific failure modes. First, the critic learns almost nothing about long correct answers if it only sees data from the untrained starting policy, because that policy rarely produces them — so they pretrain on a pool of trajectories from policies of several different strengths. Second, the newly added prediction head produces huge gradients that swamp the rest of the network, so they initialize it with deliberately small weights (standard deviation 1/(H+1)) and raise the gradient-clipping threshold for the critic from 1 to 10 so the head's large gradients do not cause the network's much smaller gradients to be shrunk too aggressively.

Online, the critic learns faster than the policy — a critic learning rate five times the actor's and two critic epochs per rollout batch versus one actor epoch — and it is used to compute token-by-token advantages via generalized advantage estimation. The policy itself is updated with the CISPO objective.

For prompt selection, they convert the critic's value at the start of a response into an estimated success probability, average it with the prompt's observed pass rate (with a prior count of 3 and a decay of 0.99 per rollout batch on the counts), and accept the prompt with probability that peaks at an estimated success rate of 0.5, with an acceptance floor (a_min = 0.3) so that very easy and very hard prompts are not excluded entirely. This selection happens before any generation, so wasted rollouts are avoided rather than discarded afterwards.

Why This Matters

The paper pushes back on the now-standard assumption that critics are too expensive and too unstable to be worth training for long-CoT reasoning. If a critic can be trained reliably, it gives finer-grained credit assignment than a single response-level score, and — the distinctive move here — it doubles as a difficulty estimator that costs nothing extra, since it is already being trained for policy learning.

Real-world applications:

• Math and science tutoring systems that need to reward correct intermediate reasoning steps rather than only final answers, where the benchmark suite here (MATH500, AIME, OlympiadBench, GPQA-Diamond) is directly representative.
• Multimodal reasoning assistants, since the 35B and 397B models are evaluated on MMMU-Pro, MathVision, and MathVista-mini, indicating the recipe carries over to vision-language mixture-of-experts settings.
• Cost-constrained training pipelines that want to move from many rollouts per prompt to one, which is where the largest reported gap appears (80.9 versus 70.4 at 35B).
• Any verifier-graded RL loop where compute is wasted generating groups of responses that get thrown away because all of them are correct or all are wrong; DAPS targets exactly that waste.

Industry relevance: the 27–40% wall-time overhead per actor gradient step is a real cost, but it buys stability where critic-free single-rollout baselines degrade after initial gains — relevant for teams training large MoE models where a collapsed run is far more expensive than a slower one.

Future Directions

• Agentic multi-turn settings. The authors explicitly leave open whether the recipe transfers to tasks where credit must be assigned across tool calls and environment turns; all experiments here are single-turn reasoning tasks with verifiable rewards.
• Seed variance. The paper states that variability across independent RL training seeds is not systematically assessed, and notes that some runs (the CISPO/DIS policy-loss variants scoring 79.8–80.2) are single-seed with different restart histories.
• Reducing critic overhead. The 27–40% per-step wall-time penalty invites work on cheaper critic inference or architectures that narrow the gap with critic-free training.
• Whether difficulty estimation can be improved beyond the value-plus-history blend. DAPS uses a fixed prior count (n_c = 3) and a fixed acceptance floor (a_min = 0.3) with a peak at 0.5; the sensitivity of these choices is not explored in the reported results.

Target Audience

Researchers and engineers working on RL post-training for LLMs, particularly those who already use GRPO-style group rollouts and are considering whether a critic is worth the complexity. It is also useful for practitioners running large mixture-of-experts training who care about run stability, and for readers interested in curriculum or prompt-selection methods that avoid a separate difficulty model. Readers new to policy-gradient methods will need background reading on PPO and generalized advantage estimation to follow the method section.

Authors’ abstract

Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.