Consensus Group Relative Policy Optimization for Text Generation

AI summary

Overview

Research area: Machine learning / natural language processing — reinforcement learning from reward-free consensus for text generation (machine translation, summarization, and social-bias mitigation).

Technical level: Intermediate. The method itself is conceptually simple (score samples against each other, then apply GRPO), but it assumes familiarity with GRPO, Minimum Bayes Risk (MBR) decoding, and policy-gradient notation. The theoretical section is Advanced.

One-sentence scope: The paper proposes C-GRPO, a reference-free training objective that distills expensive sample-and-rerank consensus decoding (specifically MBR decoding) into a single-pass policy, and evaluates it on WMT 2024 translation, XSum summarization, and the Japanese JBBQ bias benchmark.

What This Paper Is About

Many strong text-generation methods work by sampling several candidate outputs, scoring each one by how much it agrees with the other samples under a task-specific utility function, and returning the highest-scoring candidate. This works well but is expensive at inference, because every input needs multiple generations plus many pairwise comparisons. The paper's goal is to move that consensus computation from inference time into training time, so a single forward pass at test time can produce the quality that sample-and-rerank would have produced — without gold references, teacher labels, or a preference dataset.

Key Contributions

1. A reference-free distillation objective. The authors introduce Consensus Group Relative Policy Optimization (C-GRPO), which formulates the MBR consensus utility as a group-relative advantage inside GRPO. The reward is simply the average similarity of a sample to the other samples in the same group: $R(q,y_i) = \hat{u}(y_i \mid q) = \frac{1}{G}\sum_{j=1}^{G} u(y_i, y_j)$. No gold references, preference labels, or separately trained reward model are required — only a task utility function and on-policy samples.

2. Supporting theory. Through an idealized sentence-level surrogate, the paper shows that under specified assumptions the expected C-GRPO update is directionally aligned with the true expected-utility policy gradient (Theorem 3.3), and derives a non-asymptotic convergence rate of $\mathcal{O}(1/\sqrt{T})$ with a constant learning rate $\eta_t = 1/\sqrt{T}$ (Theorem 3.4).

3. Empirical validation across three task families. Experiments on WMT 2024 machine translation (En→Ja, Zh, De), XSum summarization, and JBBQ Japanese social-bias mitigation show that C-GRPO often matches or exceeds inference-time MBR decoding while using one generation instead of sample-and-rerank.

4. A demonstrated advantage over self-rewarding. C-GRPO is compared against GRPO with an LLM-as-a-judge reward and is shown to be less sensitive to judge prompts and stronger on the social-bias benchmark.

Main Findings

• Machine translation: Averaged over three WMT24 language pairs and two base models (six settings), C-Dr.GRPO scored best at 0.748 COMET and C-GRPO second at 0.719, both above MBR decoding (0.711), GRPO w/ Self-Rewarding (0.714), SFT distilled from MBR decoding (0.672), GRPO w/ Random (0.638), and the base model (0.616).

• Inference-cost comparison: With $G=32$, MBR decoding requires 32 generations and 1024 pairwise utility calls per input (an $\mathcal{O}(G^2)$ burden). C-GRPO performs no reranking or utility evaluation at test time.

• Self-rewarding is prompt-sensitive: Using a different judging prompt (Unbabel), the self-rewarding baseline's average COMET dropped from 0.707 to 0.465 across En→Ja, Zh, De and both base models.

• Summarization (XSum, Llama): Base model 0.362 ± 0.002 ROUGE-Lsum; MBR decoding 0.365 ± 0.004; C-GRPO 0.410 ± 0.009; C-Dr.GRPO 0.416 ± 0.006. For Mistral, base was 0.231 ± 0.002, MBR decoding 0.247 ± 0.002, and C-GRPO also 0.247 ± 0.004, while C-Dr.GRPO reached 0.232 ± 0.003. The other reported Mistral rows were GRPO w/ Random 0.226 ± 0.005, GRPO w/ Self-Rewarding 0.222 ± 0.010, and SFT (MBR decoding) 0.235 ± 0.002; for Llama, GRPO w/ Random was 0.329 ± 0.008, GRPO w/ Self-Rewarding 0.254 ± 0.024, and SFT w/ MBR generations 0.353 ± 0.009.

• Paired instance-level analysis: On the same 500 XSum test examples, C-GRPO beat MBR on 384 instances versus 116 for MBR, with a mean paired difference of +0.0554 and a 95% paired-bootstrap interval of [+0.0485, +0.0622]. A paired sign-flip test reported $p < 10^{-6}$, with none of 999,999 random sign flips producing a difference at least as extreme.

• Robustness across backbones: On XSum ROUGE-Lsum, C-GRPO matched or exceeded the base score for most models across families and scales (Llama, Mistral, Qwen). Clear degradation appeared only for the smallest Qwen2 variants (0.5B and 1.5B); larger Qwen2.5 models and Llama and Mistral variants consistently benefited.

• Japanese social-bias mitigation (JBBQ): Trained on 10,000 randomly selected samples and tested on 500, C-GRPO raised Mistral accuracy from 26.8% to 34.4% and Llama accuracy from 42.6% to 49.0%. MBR decoding reached 31.0% (Mistral) and 48.0% (Llama), while GRPO w/ Self-Rewarding reached 27.4% and 46.0% respectively.

• Candidate budget: C-GRPO remained effective across $G \in {4, 8, 16, 32}$ in the En-Ja translation setting (Appendix E).

• Why C-GRPO can exceed finite-sample MBR: Finite-sample MBR can only choose among candidates drawn from the fixed base policy, whereas C-GRPO reshapes the policy toward high-consensus outputs generatable in one pass. The paper reports that output-score variance decreases across checkpoints as quality improves (Appendix F).

Methodology in Plain English

The researchers take an existing reinforcement learning recipe for language models, GRPO, and swap out what it is rewarded for. In standard GRPO, each sampled output is scored by a reward or preference model, and the advantage compares that score to the group average. C-GRPO replaces that external score with a consensus score: how similar a sample is, on average, to all the other samples drawn for the same prompt. This is the Monte Carlo estimate of MBR's expected utility, but computed as an internal, group-relative quantity. The only ingredient needed is a similarity/utility function; the paper uses BLEURT for all tasks.

Because the reward is a within-group comparison rather than an absolute score against labeled data, no gold references, teacher labels, or collected preference datasets are needed, and no separate reward model must be trained. At test time the model just generates once — the reranking step is gone.

For the theory, the authors simplify to a sentence-level surrogate where the whole output sequence is treated as a single action, all sampled outputs are assumed to have the same length, and the inverse group standard deviation is treated as a scalar preconditioner independent of the gradient direction. Under L-smoothness, bounded variance, support/absolute-continuity, and this independence condition (plus $G \geq 2$), they prove the expected update points in the same direction as the expected-utility gradient, then derive the $\mathcal{O}(1/\sqrt{T})$ rate.

The experiments compare against several baselines: the base instruction-tuned models, MBR decoding, GRPO with a random Gaussian reward $\mathcal{N}(0,1)$, GRPO with a self-judging LLM-as-a-judge reward, and (where references are available) SFT, DPO, and Best-of-N. Evaluation uses metrics that are not always the training utility: COMET for translation, ROUGE-Lsum for summarization, and accuracy for JBBQ.

Why This Matters

Impact on research. The paper connects two previously separate lines of work — consensus/MBR decoding and GRPO-style policy optimization — and shows that the sample-and-rerank paradigm's decision rule can be compiled into model weights instead of executed at test time. It also identifies a concrete failure mode of self-rewarding: judge prompt sensitivity and inherited bias. The theoretical result gives a formal link between a group-relative advantage and an expected-utility objective, which may be reusable for other consensus-style utilities beyond MBR.

Real-world applications:

• Machine translation serving, where latency budgets make 32-sample reranking infeasible but translation quality matters.
• Summarization of news articles, the XSum setting studied here, where single-pass generation is a deployment requirement.
• Bias-aware question answering and other sensitive social applications, given the JBBQ results on disability status, gender identity, physical appearance, and sexual orientation.
• General LLM serving pipelines that currently pay for Best-of-N or MBR reranking behind an API.

Industry relevance. The inference-cost argument is the practical hook: replacing 32 generations plus 1024 pairwise utility calls per input with a single generation directly reduces serving cost and latency. The reference-free property is also relevant to teams that lack curated preference data or high-fidelity reward models, since only a task utility function is needed.

Future Directions

• Utility quality and candidate diversity. The limitations section notes that if the utility function is imperfectly aligned with human judgment, or the base model cannot produce sufficiently diverse high-quality candidates, C-GRPO may distill those shortcomings into the policy.
• Broader validation. The authors state that their experiments rely primarily on automatic metrics and call for broader validation with larger model scales and longer-form generation (the text is truncated at this point).
• Theory beyond the idealizations. The analysis treats $(\sigma_{\mathcal{G}})^{-1}$ as an approximately independent scalar preconditioner, and the authors note Dr. GRPO's analysis does not require this independence assumption — a tighter analysis of the correlated normalization is an open avenue.
• Small-model behavior. The one clear degradation pattern was on the smallest Qwen2 variants (0.5B and 1.5B), raising the question of what model capacity is needed for consensus distillation to help rather than hurt.

Target Audience

Researchers and practitioners working on reinforcement learning for language models, decoding methods, or inference-efficient text generation. It is most useful to readers already comfortable with GRPO, PPO-style policy gradients, and MBR/consensus decoding. Applied engineers responsible for translation or summarization serving will find the cost-versus-quality argument directly actionable, while those interested in reference-free alignment and bias mitigation will find the JBBQ comparison with self-rewarding the most relevant part.

Authors’ abstract

Many strong decoding methods for text generation follow a sample-and-rerank paradigm: they draw multiple candidates, score each under a utility (reward) function using consensus across samples, and return the best one. Although effective, these methods incur high computational costs during inference due to repeated sampling and scoring. Prior attempts to amortize inference-time computation typically rely on gold references, teacher labels, or curated preference data, increasing dataset construction effort and the demand for high-fidelity reward models. We propose Consensus Group Relative Policy Optimization (C-GRPO), which distills Minimum Bayes Risk (MBR) decoding into training by formulating the consensus utility as a group-relative objective within GRPO. C-GRPO requires only a utility function and policy samples, without gold references or explicit preference labels. Under ideal conditions, we show that the objective function of C-GRPO is directionally aligned with the gradient of the expected-utility objective underlying MBR decoding, leading to a convergence guarantee. Experiments on machine translation (WMT 2024) and text summarization (XSum) demonstrate that C-GRPO successfully achieves performance comparable to MBR decoding without the associated inference-time overhead, while outperforming reference-free baseline methods.