Skip to content
AI.info

Advanced techniques

Actor–Critic Methods and Proximal Policy Optimization

Connect policy gradients, critics, advantages, on-policy data, and PPO’s clipped objective in a practical deep-RL workflow.

By the end you can

From “which action worked?” to a trainable policy update

Policy-gradient methods adjust a policy toward actions associated with higher return. The raw return signal is noisy. Outcomes depend on environment randomness, on long action sequences, and on other choices made in the same episode.

An actor–critic system separates two roles. The actor represents the policy that chooses actions. The critic estimates expected future return from a state, or from a state–action pair. That estimate becomes a baseline for judging whether the sampled action performed better or worse than expected.

The separation reduces variance and introduces critic error. A biased or unstable critic can misdirect the actor. So value loss, explained variance, advantage statistics, and policy behavior have to be read together, not one at a time.

The critic is a learned yardstick for the actor’s sampled decisions, not an oracle.

Visual

The actor–critic training cycle

The loop alternates between collecting fresh behavior and improving value and policy estimates. OpenAI Five is that loop written out at full scale, with every box documented.

On 13 April 2019 a reinforcement-learning system beat the Dota 2 world champions OG, 2–0. OpenAI’s paper opens with the result: “On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game.” The algorithm was PPO, which the paper describes as “a variant of advantage actor critic”, using Generalized Advantage Estimation.

The numbers give the five boxes a scale. One update consumed an effective batch of 2,949,120 timesteps. Training ran across up to 1,536 GPUs and consumed 770 ± 50 PFlops/s·days over 180 days. Box one — running the current policy — was itself a deliberate mixture. Rollout workers played the latest policy against itself 80% of the time, and against older policies 20% of the time. That is what a team does when it knows the data distribution moves with the policy that produced it.

The public evaluation deserves the same care the loop demands. In the OpenAI Five Arena of 18–21 April 2019 the system played 3,193 teams over 7,257 games and won 99.4%. The paper also records that 3,140 of its 7,215 wins were games humans abandoned. The headline number and the games behind it are not the same claim.

FigureProcess · 5 steps
  1. 1

    Run the current policy

    Collect trajectories of states, actions, rewards, and terminations.

  2. 2

    Estimate returns and advantages

    Use rewards and the critic to score sampled actions relative to expectation.

  3. 3

    Update the critic

    Reduce error between predicted values and return targets.

  4. 4

    Update the actor

    Increase probability of advantageous actions and decrease disadvantageous ones.

  5. 5

    Measure policy movement

    Track KL divergence, clipping, entropy, and behavioral change.

On-policy algorithms need new rollouts because the data distribution depends on the current policy: OpenAI Five still drew 80% of its self-play from the latest policy.

Comparison

What each signal tells you

Confusing these quantities makes RL debugging nearly impossible. And the advice usually attached to them — “trades bias against variance”, “should be normalized cautiously” — has been measured rather than left to taste.

More than 50 design choices were implemented inside a single on-policy actor–critic framework, and then the sweep ran: “We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.” Andrychowicz and eleven colleagues published that in 2021.

Their verdicts land directly on the three columns below. For the advantage estimate: GAE with λ = 0.9. For PPO’s clipping threshold: start at 0.25. And the two halves of the usual normalisation advice come apart under measurement. Value-function normalisation “influences the performance very strongly”, while “per-minibatch advantage normalization (C67) seems not to affect the performance too much”.

One of the two knobs you were told to treat carefully is doing most of the work. The other is close to invisible at 250,000 agents.

FigureComparison · 3 columns

Return

The discounted reward accumulated from a point in the trajectory.

  • Sampled from experience
  • High variance
  • Depends on horizon and discounting
  • Defines what the agent ultimately values

Value estimate

The critic’s prediction of expected return from a state.

  • Acts as a baseline
  • Can be biased
  • Trained by regression-like objectives
  • Useful for bootstrapping

Advantage estimate

How much better an action performed than the baseline expectation.

  • Guides the policy update
  • Often estimated with temporal-difference methods
  • Trades bias against variance
  • Should be normalized cautiously

What PPO clipping does—and does not do

PPO compares the probability of each sampled action under the new policy with its probability under the policy that generated the data. The ratio measures how much the update changes the likelihood of that action. The clipped surrogate objective removes some of the incentive to push that ratio beyond a chosen range. This makes large, destructive policy changes less attractive, and it allows several minibatch epochs over the same on-policy rollout.

Whether the clip actually holds the update inside that range is a measurable question, and it has been measured. Probability ratios were logged during training on MuJoCo Humanoid-v2, and the maximum ratios consistently exceeded the 1+ε clipping bound. Engstrom and six colleagues reported it in 2020: “First, we note that all three algorithms fail to maintain a ratio-based trust region, despite PPO and PPO-M being trained directly with a ratio-clipping objective.” Wang and two colleagues reached the same conclusion independently in 2019, showing that PPO “could neither strictly restrict the probability ratio as it attempts to do nor enforce a well-defined trust region constraint”.

So the clipped objective is an incentive, not a fence. It does not guarantee that the whole policy stays within a fixed distance. It does not guarantee monotonic improvement. Implementations monitor approximate KL divergence, and often stop an update early when policy movement gets excessive. PPO is used widely because it is simple to get right and performs well enough. It did not solve the instability or the sample-efficiency problem of reinforcement learning.

Two independent groups measured ratios past the clip bound: clipping discourages large updates, it does not bound them.

Case

Ten runs, one configuration, two different conclusions

Take ten runs of one configuration, change nothing but the random seed, and split them down the middle. That is the cheapest possible test of how much a deep-RL result depends on nothing, and somebody ran it: “We perform 10 experiment trials, for the same hyperparameter configuration, only varying the random seed across all 10 trials. We then split the trials into two sets of 5 and average these two groupings together”. The environment was HalfCheetah-v1. The algorithm was TRPO. The only difference between the two curves was the seed.

The averages came out far enough apart to be significant. “The average 2-sample t-test across entire training distribution resulted in” t = −9.0916 and p = 0.0016. Henderson and five colleagues published that in 2018. Two halves of one experiment would have been published as two findings.

Three years later the Atari 100k benchmark got the same treatment. Agarwal and four colleagues recomputed median scores from “100,000 different sets of N runs subsampled uniformly with replacement from 100 runs”, and found “substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis”. The runs had not changed. The confidence placed in them had.

Example

A dashboard for PPO training

No single scalar tells you whether the agent is learning the intended behavior. And the dashboard is not improvised.

A public reproduction of PPO needed a checklist of 37 implementation details to match the original results: 13 core, 9 Atari-specific, 9 for continuous-action robotics, 5 for LSTM and 1 for MultiDiscrete action spaces. It appeared on the ICLR 2022 Blog Track. Along the way it published reference behaviour for the standard debug variables, which is what turns a logged curve into a diagnostic.

  • Episodic return and task success: Track distributions, not only the mean, because rare failures can matter — and because ten identical runs split into two groups of five can separate at p = 0.0016.
  • Policy loss and value loss: Divergent behavior can reveal a critic that cannot keep up with the actor.
  • Explained variance: Indicates how much of return variation the critic captures, though it is not a complete quality measure. Value-function normalisation is the setting most worth checking here: across 250,000 agents it “influences the performance very strongly”.
  • Entropy: Low entropy may indicate premature policy collapse; high entropy may indicate persistent indecision.
  • Approximate KL and clip fraction: Both have published definitions and a published reference range. The 2022 reproduction defines clipfrac as “the fraction of the training data that triggered the clipped objective” and computes approx_kl as (-logratio).mean(). Its guidance is explicit: “We have generally found the approx_kl stays below 0.02, and if approx_kl becomes too high it usually means the policy is changing too quickly and there is a bug.” Stable-Baselines3, maintained independently at DLR, logs the same two quantities, computing clip_fraction as the mean of |ratio − 1| > clip_range.
  • Reward components and behavior videos: Detect reward hacking that aggregate return can hide.

Key idea

A stable optimizer can still optimize the wrong reward

PPO can produce smooth training curves while the agent exploits a loophole in the reward, the simulator, or the termination rule. Stabilizing policy updates does not align the objective with human intent. That failure has been given a shape and a curve.

Hold one “gold-standard” reward model fixed as ground truth. Optimise a policy against a proxy reward model. Then fit how the gold score moves as optimisation proceeds: R_RL(d) = d(α_RL − β_RL·log d) for reinforcement learning, and R_bon(d) = d(α_bon − β_bon·d) for best-of-n, with d defined as the square root of D_KL(π ‖ π_init). Put that distance from the initial policy on the x-axis and the two curves part company. The proxy keeps climbing. The true objective turns over.

Gao and two colleagues summarised it in a line: “Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law.” Coste and three colleagues reproduced the setup with PPO in 2024, added 25% label noise, and found that ensemble-based conservative optimisation always reduced the overoptimisation. Log reward components separately. Review representative trajectories.

Environment bugs are especially dangerous. The agent searches aggressively for repeatable advantages. Test resets, observation timing, action bounds, episode truncation, and reward scaling before you interpret algorithm performance.

Optimization stability cannot compensate for a misspecified reward or environment — the proxy curve keeps rising after the real one turns over.

Analogy

Updating a playbook after each match

After each set of matches the coach revises the team’s playbook. The actor is the playbook. The critic estimates how promising each game situation is. The coach increases the use of plays that worked better than expected.

PPO resembles limiting how radically the playbook changes from one review cycle to the next. The match data came from the old playbook. Large updates make that evidence less representative of the new behavior. That is one reason on-policy data goes stale so quickly.

PPO reuses recent experience cautiously because policy changes alter the data-generating process.

Steps

A reproducible PPO experiment

RL variance demands more discipline than one successful run. Step 2 — establishing baselines — is the step usually waved through.

Here is what it looks like when it is not. PPO’s clipped step was ablated against its “code-level optimizations” on three MuJoCo tasks, with at least 80 agents trained for each estimate and 1000-sample bootstrap 95% confidence intervals. On Hopper-v2, PPO scored 2513 [2391, 2632] against PPO-M’s 2142 [2008, 2279]. TRPO scored 2043 [1948, 2136] against TRPO+’s 2466 [2381, 2549]. Average code-level improvement: 421. Average algorithmic improvement: 99. A variant with no clipping at all reached 831 [798, 869] on Humanoid-v2, against 806 [785, 827] for full PPO.

Engstrom and six colleagues drew the conclusion themselves in 2020: “We go on to test the importance of code-level optimizations in agent performance, and find that PPO’s marked improvement over TRPO (and even stochastic gradient descent) can be largely attributed to these optimizations.” The credit for PPO’s reputation went, on measurement, mostly to the code around the algorithm. That is the price of a real baseline. It is also why the steps below insist on distributions rather than a best run.

FigureProcess · 6 steps

  1. 1. Validate the environment

    Use scripted policies and unit tests for rewards, resets, and terminations.

  2. 2. Establish simple behavior baselines

    Compare random, heuristic, and supervised or imitation policies where possible.

  3. 3. Run multiple seeds

    Report distributions and learning curves rather than the best seed.

  4. 4. Log update diagnostics

    Track KL, clipping, entropy, value error, and advantage statistics.

  5. 5. Inspect trajectories

    Review success, failure, and high-reward episodes for reward exploits.

  6. 6. Test robustness

    Vary initial states, dynamics, observation noise, and reward scales.

Example

Forensics for a suspicious PPO update

A stable-looking reward curve can hide policy collapse, critic error, or exploitation of the reward definition.

  • Large KL jump: Check learning rate, number of epochs, clipping fraction, and stale rollout reuse — and note that the reference implementations already expect this. OpenAI’s Spinning Up PPO defaults to clip_ratio = 0.2 with target_kl = 0.01, and stops gradient steps once the approximate KL exceeds 1.5 × target_kl. Its documentation gives the grounds: “While this kind of clipping goes a long way towards ensuring reasonable policy updates, it is still possible to end up with a new policy which is too far from the old policy, and there are a bunch of tricks used by different PPO implementations to stave this off.” Stable-Baselines3 ships the same switch as target_kl, default None, documented as being there to “Limit the KL divergence between updates, because the clipping is not enough to prevent large update”.
  • Value loss falls while return stalls: Inspect critic target quality and whether the policy receives useful advantages.
  • Entropy collapses early: Confirm that exploration incentives and action masking are implemented as intended.
  • Reward rises but behavior worsens: Review trajectories for reward hacking and missing constraints — this is the regime Gao and colleagues fitted, where the proxy score climbs as KL from the initial policy grows and the gold score turns over.
  • Explained variance becomes negative: Investigate value normalization, bootstrapping, termination handling, and observation leakage.
  • High seed variance: Report the distribution of outcomes and compare with simpler policies or behavior cloning; ten TRPO trials that differed only in seed split into two significantly different averages at t = −9.0916, p = 0.0016.

Key takeaways