Group Adaptive Clipping Policy Optimization

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for large language model post-training, specifically group-relative policy optimization methods such as GRPO, GSPO, and DAPO.

Technical level: Advanced. The paper assumes familiarity with PPO-style clipped surrogates, importance-sampling ratios, trust-region policy optimization, and reverse KL regularization.

Scope: The paper diagnoses a specific flaw in fixed importance-sampling (IS) ratio clipping and proposes a closed-form, advantage-dependent clipping schedule that is a drop-in replacement for the clip boundary in existing group-relative algorithms.

What This Paper Is About

Group-relative RLVR methods such as GRPO and GSPO apply one fixed IS-ratio clipping boundary to every rollout, even though a rollout's

Authors’ abstract

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.