M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization

AI summary

Overview

• Research area: Post-training of large language models (LLMs) for reasoning via self-supervised reinforcement learning with verifiable reward (RLVR), i.e., training without ground-truth answers using the model's own outputs as reward signal.
• Technical level: Advanced. The paper assumes familiarity with policy gradient methods, GRPO (Group Relative Policy Optimization), advantage normalization, momentum contrastive learning, and trajectory-level entropy.
• Scope in one sentence: The paper diagnoses a "policy collapse" failure mode in self-supervised RLVR for LLMs and proposes two fixes — a momentum-anchored policy optimization framework (M-GRPO) and an interquartile-range (IQR) trajectory entropy filter — evaluated on Qwen3-4B-Base trained on the MATH dataset without labels.

What This Paper Is About

Self-supervised RLVR lets an LLM improve its reasoning by generating its own pseudo-labels (for example, by majority voting over sampled answers), avoiding costly human annotations. The authors show that these methods work briefly and then crash: training reward and validation accuracy rise, then degrade sharply or progressively during long-horizon training — a failure they call "policy collapse." The goal is to stabilize this training so that the model keeps improving instead of needing a manually cherry-picked checkpoint before the crash, and to simultaneously prevent the early collapse of policy entropy that produces a prematurely confident, suboptimal policy.

Key Contributions

1. Diagnosis of policy collapse. The authors identify and document a critical instability in self-supervised RLVR under long-horizon training, and show that scaling the number of rollouts (G) only delays, but does not prevent, the collapse.
2. M-GRPO (Momentum-Anchored Group Relative Policy Optimization). A framework that maintains a momentum model whose parameters are an exponential moving average of the current policy's parameters. Rollouts from this slowly evolving model are added to the majority-voting pool used to construct the pseudo-ground-truth answer, providing a stable training target.
3. IQR-based trajectory entropy filter. A dynamic filtering method that computes trajectory-level entropies for a prompt's rollouts, derives the interquartile range, and discards low-entropy outliers below the threshold Q1 − k·(Q3 − Q1), preserving policy diversity and preventing premature convergence.
4. Extensive evaluation. Experiments on multiple reasoning and code benchmarks showing that M-GRPO stabilizes training and that the final checkpoint outperforms the collapsed final checkpoint of the SRT baseline and is competitive with, and frequently surpasses, the manually selected best SRT checkpoint.

Main Findings

• Policy collapse is real and reproducible. In the authors' reproduction of SRT (Shafayat et al., 2025) and Intuitor (Zhao et al., 2025) on the MATH training split, training reward rises initially and then "precipitously or progressively crashes," accompanied by degradation in validation accuracy on the MATH test split. Figure 5 shows this for SRT-MV-Qwen2.5-3B, SRT-MV-Qwen2.5-7B, and Intuitor-MV-Qwen2.5-3B.
• Scaling rollouts delays but does not prevent collapse. In Table 1, larger rollout counts improve SRT's peak accuracy — the paper states that scaling from G=16 to G=64 raised best MATH500 accuracy from 76.15% to 79.20% — but the collapse still occurs (Figure 1). All Table 1 numbers are from manually selected best checkpoints chosen before collapse.
• Entropy collapse accompanies the instability. During early training of a standard SRT model, policy entropy drops sharply, producing an overly confident policy (left of Figure 2). M-GRPO maintains a higher entropy that decreases more gradually (right of Figure 2).
• M-GRPO trains stably without checkpoint cherry-picking. M-GRPO sustains an improving, stable reward throughout training across Qwen3-4B-Base, Qwen3-1.7B-Base, and Qwen3-8B-Base backbones, translating into consistently high validation accuracy with no observed degradation (Figure 4).
• Final-checkpoint performance. In Table 2, the collapsed SRT final checkpoint (SRT_Final) drops to 47.50% on MATH500 (from a best of 79.20%), 7.50% on AIME24, 8.75% on AIME25, 28.54% on GPQA Diamond, 25.89% on GPQA, and 16.12% on LiveCode. The M-GRPO+IQR_Final row reports 79.75% on MATH500, 14.58% on AIME24, 14.17% on AIME25, 39.65% on GPQA Diamond, and 35.49% on GPQA (no LiveCode value is given for that row).
• Comparison against the best SRT checkpoint. SRT_Best reaches 79.20% (MATH500), 12.50% (AIME24), 11.67% (AIME25), 38.26% (GPQA Diamond), 35.04% (GPQA), and 19.69% (LiveCode). The paper reports absolute improvements for M-GRPO of +2.92 points on AIME24, +5.05 on GPQA, and +7.43 on LiveCode, and states that M-GRPO achieves a final GPQA accuracy of 40.09% versus SRT's peak of 35.04%. Note that the 40.09% prose figure does not match the 35.49% GPQA value shown for M-GRPO+IQR_Final in Table 2; the largest GPQA value appearing in Table 3 is 38.39% at G=256.
• Rollout scaling of M-GRPO plateaus. Table 3 shows M-GRPO+IQR improving as G scales from 8 to 32 (MATH500: 77.60% → 79.75%; AIME24: 11.25% → 14.58%; mbpp: 68.60% → 70.60%), with gains plateauing beyond that — at G=256, results are 79.50% (MATH500), 16.67% (AIME24), 14.17% (AIME25), 40.66% (GPQA Diamond), 38.39% (GPQA), 55.08% (MMLU-pro), and 70.40% (mbpp).
• Baseline starting point. The unmodified Qwen3-4B-Base scores 61.50% on MATH500, 0.83% on AIME24, 5.00% on AIME25, 34.41% on GPQA Diamond, 29.91% on GPQA, 9.61% on LiveCode, 51.38% on MMLU-pro, and 63.40% on mbpp.
• Table labeling caveat. In Table 2, two consecutive rows are both labeled "SRT_Final"; given the accompanying discussion, the second of these appears intended as an M-GRPO condition, but the paper as presented does not distinguish them by name.

Methodology in Plain English

The method builds on GRPO, which trains a policy by comparing groups of sampled answers for the same prompt and normalizing rewards across the group to reduce gradient variance. Two models are maintained:

1. The current policy (the model being trained).
2. A momentum model, which is never updated by gradients. Its weights are an exponential moving average of the current policy's weights, following the rule π_θk ← m·π_θk + (1−m)·π_θq, with m ∈ [0,1) (e.g., 0.99), so it changes slowly and behaves like a stable reference. This design is adapted from momentum contrast in self-supervised visual representation learning.

For each prompt, the current policy samples M responses and the momentum model samples N responses, giving G = M + N total rollouts. These are pooled, and a pseudo-ground-truth answer is selected by majority voting — the answer agreeing with the most other answers in the pool. Including the momentum model's rollouts is what stabilizes the pseudo-label, since labels derived purely from the fast-changing current policy are noisy. Rewards are binary (1 for agreeing with the pseudo-ground-truth, 0 otherwise), and advantages are computed only over the current policy's M responses using the standard GRPO normalization.

Before voting, the trajectory-level entropy of each of the G rollouts is computed. The first and third quartiles (Q1, Q3) of these entropies define the interquartile range, and trajectories with entropy below Q1 − k·(Q3 − Q1) are discarded, with k = 0.75 used in all experiments. This data-adaptive cutoff is preferred over a static threshold (such as removing the bottom 10%) because entropy distributions shift both across prompts and over the course of training; at the start of training nearly everything is high-entropy, so a static rule would be ineffective.

The policy is then updated to maximize the advantage-weighted log-likelihood of its responses (Eq. 3), and the momentum model is updated by the moving-average rule. Training used VeRL on 8 × NVIDIA H200 GPUs, with a train batch size of 8 questions, maximum prompt length 512, maximum response length 3072, clip ratio 0.2, AdamW (β1 = 0.9, β2 = 0.999, ε = 10^-8), learning rate 1e-06, cosine warmup with 0.1 warmup steps ratio, KL loss coefficient 0.005, training temperature 1.1, and evaluation temperature 0.8. For M-GRPO, the number of momentum rollouts is N = G/4. AIME and GPQA Diamond problems were sampled 8 times and averaged; the remaining benchmarks were sampled once.

Why This Matters

The paper argues that self-supervised RL is attractive precisely because it avoids annotation cost and reward-model infrastructure, but that this promise is undermined if training collapses and requires manual checkpoint selection before deployment. M-GRPO's claimed contribution is a training recipe that reaches comparable or better accuracy at the final checkpoint than a hand-picked peak checkpoint of the baseline, which matters for any pipeline that must run continuously without a human watching for the crash.

• Impact on research: It reframes self-supervised RLVR instability as a missing-stable-target problem rather than a purely data or reward-design problem, and it imports momentum-based anchoring from self-supervised vision representation learning into LLM policy optimization. It also documents that rollout scaling, a common remedy, is not sufficient on its own.
• Real-world applications:
  • Post-training reasoning models on domain corpora where answers are unavailable but self-consistency is a usable signal (e.g., internal documents, specialized mathematics or science problem sets).
  • Long-running RL fine-tuning pipelines that need stable, unattended training rather than checkpoint triage.
  • Code generation improvement using unlabeled problem sets, given the reported gains on LiveCode and mbpp.
  • Improving smaller, cheaper models — the method is demonstrated on base models in the Qwen3 family, including a 1.7B variant in Figure 4.
• Industry relevance: The main practical appeal is eliminating the "train until it breaks, then roll back" workflow. Stability over long horizons reduces compute waste from collapsed runs, removes a manual selection step that is hard to automate, and makes self-supervised RL more viable where labeled preference data or programmatic verifiers are unavailable or expensive.

Future Directions

• Determine whether the momentum anchor alone is sufficient. The paper presents M-GRPO and the IQR filter as two separate contributions addressing two separate failure modes (policy collapse and entropy collapse), but does not report an ablation isolating each component's individual effect in the provided content.
• Explain the plateau. M-GRPO+IQR improves from G=8 to G=32 but gains only marginally up to G=256; what limits further scaling, and whether larger momentum pools or alternative pseudo-label schemes could extend the gains, is an open question.
• Reconcile reported numbers and extend the checkpoint analysis. Some prose figures (notably the 40.09% GPQA claim) differ from the table values, and the duplicated "SRT_Final" row label in Table 2 needs clarification; resolving these would make the stability claim easier to verify.
• Test beyond the studied settings. All results use Qwen3 backbones (4B-Base in the main table, with 1.7B, 4B, and 8B in the stability figure) trained on MATH without ground truth; behavior on other model families, on longer training horizons, and on domains where majority voting is a weaker proxy for correctness is not reported.
• Quantify cost. The paper does not report wall-clock time, token counts, or total compute budget, so the extra cost of running a momentum model (N = G/4 rollouts from it) relative to SRT is not stated.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning from verifiable rewards, and self-supervised reasoning improvement. The paper is most useful to those who have run or plan to run RLVR-style training without ground-truth labels and have encountered late-stage degradation, and to practitioners who need training recipes that do not depend on manually selecting a checkpoint. Readers need a working understanding of policy gradient methods and GRPO-style advantage normalization to follow the methodology; the diagnostic findings about collapse, however, are accessible to a broader machine learning audience.

Authors’ abstract

Self-supervised reinforcement learning (RL) presents a promising approach for enhancing the reasoning capabilities of Large Language Models (LLMs) without reliance on expensive human-annotated data. However, we find that existing methods suffer from a critical failure mode under long-horizon training: a "policy collapse" where performance precipitously degrades. We diagnose this instability and demonstrate that simply scaling the number of rollouts -- a common strategy to improve performance -- only delays, but does not prevent, this collapse. To counteract this instability, we first introduce M-GRPO (Momentum-Anchored Group Relative Policy Optimization), a framework that leverages a slowly evolving momentum model to provide a stable training target. In addition, we identify that this process is often accompanied by a rapid collapse in policy entropy, resulting in a prematurely confident and suboptimal policy. To specifically address this issue, we propose a second contribution: an adaptive filtering method based on the interquartile range (IQR) that dynamically prunes low-entropy trajectories, preserving essential policy diversity. Our extensive experiments on multiple reasoning benchmarks demonstrate that M-GRPO stabilizes the training process while the IQR filter prevents premature convergence. The combination of these two innovations leads to superior training stability and state-of-the-art performance.