Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

AI summary

Overview

Research area: Natural language processing — reinforcement-learning post-training of large language model recommenders (GRPO variants for conversational recommendation).

Technical level: Intermediate. The paper assumes familiarity with GRPO-style policy-gradient training, advantage normalization, and recommendation metrics such as Recall@K, but its central argument is stated in plain terms.

One-sentence scope: The paper shows that when a recommendation request has several human responders who each name different items, Rank-GRPO's practice of making one training row per responder turns legitimate answer variation into reward noise, and that scoring each generated list once against the union of all responders' items removes most of that noise and cuts the RL compute needed to reach a given quality.

What This Paper Is About

Online recommendation requests typically draw several replies, each naming a different set of titles. The state-of-the-art trainer, Rank-GRPO, turns every (request, responder) pair into a separate training example and rewards a generated list only against the one responder that its row happened to draw, so the reward largely records which reply was drawn rather than how good the list is. The paper's goal is to change the training unit so that RL trains on the request rather than on each answer, while keeping the rest of the Rank-GRPO/GRPO machinery unchanged.

Key Contributions

1. Diagnosis of responder noise. The authors quantify how much of Rank-GRPO's advantage signal is driven by the responder draw: on Reddit-v2 this share is 85% (66% on Reddit-Books, 79% on Reddit-Music), and in more than half of comparisons between the eight sampled lists every list receives the same score, so those groups teach the model nothing.
2. ReMaRk (Responder-Marginalized Rank-level RL). A method that scores each rank position of a list once against the union of all the request's responders' titles (a bounded 0/1 reward) and samples each request equally instead of once per responder. Nothing else in Rank-GRPO changes.
3. Theory for why averaging works and why a 0/1 reward is kept. Proposition 1 shows that averaging the reward over the request's responders is a Rao–Blackwellization: the expected update is unchanged and a variance term disappears, with equality when every generated item was named by all or none of the responders (in particular when there is one responder). Proposition 2 shows that under GRPO's per-group standard-deviation scaling, a fractional averaged reward can amplify one responder's vote by up to a factor of nₓ, whereas 0/1 rewards stay bounded.
4. Empirical validation across backbones, datasets, and a non-recommendation task. Seven backbones and three datasets for recommendation, plus ProtoQA (about 100 people answer each question), where the same joint-reward rule gains +3.67 [+2.64, +4.69] recall@10 at 250 steps over three seeds.

Main Findings

• Same quality for about half the RL compute. In all ten backbone and dataset settings, ReMaRk reaches the 2,000-step validation R@20 of its baseline in 25 to 53% of the steps, and leads by 0.8 to 1.6 points at 2,000 steps. In the per-setting table, steps-to-match range from 0.25 to 0.53; on Reddit-v2 with Qwen3.5-2B it is 0.52 [0.42, 0.64], on Reddit-Books 0.48, on Reddit-Music 0.33.
• Never significantly behind at the plateau, and sometimes ahead. Trained until neither arm improves, ReMaRk is never significantly behind on test, and in four of seven pairs it reaches Rank-GRPO's best with 18 to 63% of the GPU-hours. The exception is stated plainly: at 9B and on Reddit-Books, Rank-GRPO's validation peak ends 0.08 to 0.36 points higher (not significant).
• Fixed-budget lead on every backbone and dataset. With the same SFT start, length control and 2,000 RL steps, ReMaRk is ahead on all seven backbones and three datasets — for example +1.57 [1.10, 2.06] per-row R@20 over the better per-responder arm on Reddit-v2 with Qwen3.5-2B, and +0.78 [0.38, 1.17] on Reddit-Music. With batch scaling instead of group scaling it still leads by +0.80 to +1.47 points.
• GPU-hour-matched comparisons. On Reddit-v2, ReMaRk with half of Rank-GRPO's steps is level at 2B (+0.35 [−0.07, 0.77], 7 vs 13 GPU-hours) and ahead at 4B (+0.87 [0.42, 1.33], 14 vs 26 GPU-hours) and 9B (+0.60 [0.16, 1.10], 17 vs 33 GPU-hours); at 2B and 4B it also beats Rank-GRPO trained twice as long (2k vs 4k steps).
• Pooling the responders, not the aggregate, is the source of the gain. An averaged reward with per-group scaling is statistically level with ReMaRk over three seeds (−0.08 [−0.45, +0.29]), so the benefit comes from pooling the responders rather than from using a fractional mean.
• Exact Rao–Blackwellization inside Rank-GRPO also helps, but less. Averaging Rank-GRPO's own per-responder advantages at one computation per responder gains +0.93 [+0.83, +1.04] over three seeds, and ReMaRk stays above it, significantly at 2B (+0.63 [0.23, 1.00] at 2B, +0.31 [−0.07, 0.70] at 4B).
• The gain tracks the noise removed. Under batch scaling, averaging the reward over more responders raises validation recall with the share of noise removed, by 1.4 points from none to all, with a slope of +1.38 [+0.97, +1.78] (two seeds); per-group scaling erases that slope. The gain is positive for every number of responders and largest on single-responder requests, where it comes from request sampling rather than the reward.
• Stronger baselines do not close the gap. Doubling Rank-GRPO's lists per request (G=16) leaves ReMaRk ahead by +1.81 [1.35, 2.29] at 2B; dynamic sampling by +1.34 [0.92, 1.74]; offline SFT on pooled positives by +6.59 [5.90, 7.26] and DPO on pooled positives by +5.45 [4.87, 6.02].
• Length control does not explain the gain. With the length-shaping weight matched to Rank-GRPO's, ReMaRk stays +1.18 [+0.73, +1.61] ahead (+1.45 [+1.01, +1.88] without matching).
• Blind LLM judges agree and precision does not suffer. Two judge families prefer 9B ReMaRk's lists to Rank-GRPO's (share of pairwise wins 0.56 [0.52, 0.61] for GPT-5.6 and 0.58 [0.54, 0.63] for Gemini 3.1 Pro) and rate the 4B plateau pair level; precision@20 differences are positive but mostly small. The union reward does not merely buy consensus: 92 to 95% of test union items are named by one responder, and ReMaRk recalls more of them in every setting (2.1 to 2.4 points at 2B to 9B, 0.8 on Reddit-Books, 1.3 on Reddit-Music).
• On Llama-3.2-3B, 2,000 ReMaRk steps come within 0.09 points of the released Rank-GRPO checkpoint trained for 15.8k steps (23.56 vs 23.65).
• The principle transfers beyond recommendation. On ProtoQA, where about 100 people answer each question, the union gains +3.67 [+2.64, +4.69] per-respondent recall@10 over per-respondent rows at 250 steps (three seeds), exact Rao–Blackwellization about half as much (+1.94), and the batch-scaled mean is not significantly better. The union reaches the per-respondent arm's best validation recall in 42% of the steps.
• Base models show limited memorization of the answers. A few-shot probe names at least one responder item from the request alone for 5.8%, 12.6% and 18.6% of Reddit-v2 test requests (2B, 4B, 9B), and the gain holds both on those requests and the rest; on Reddit-Books and Reddit-Music the base models recall 2 to 13% of test requests.

Methodology in Plain English

Each Reddit request has several responders, each naming a few items; in the datasets, 88.0% (Reddit-v2), 95.2% (Reddit-Books) and 95.9% (Reddit-Music) of named items are named by only one person. Rank-GRPO converts each (request, responder) pair into a training row, and a generated list is rewarded only for titles that the drawn responder named. The authors argue this is like grading an exam against one randomly chosen valid answer key.

Their fix has two parts. First, the reward: instead of scoring against one responder, score each position against the union of all responders' titles — an item named by one responder counts the same as an item named by all of them, keeping the reward at 0 or 1. Second, the sampling: draw requests uniformly rather than once per responder, so requests with many responders are no longer seen many more times than requests with one.

They retain GRPO's per-group advantage normalization (dividing by the group's standard deviation at each rank position), which is why they keep the 0/1 reward: for binary rewards the group scale is shown to stay within a factor 1.51 of a constant at G=8, while a fractional average on a grid of spacing 1/nₓ can amplify a single responder's vote by up to nₓ. They note that without a length control the averaged-reward variant diverged on Llama-3.2-3B (KL rose from 0.6 to 47 after about 850 steps), and that this describes a worst case rather than a typical outcome.

All arms start from the same SFT checkpoint and use Rank-GRPO's hyperparameters (G=8, 48 requests per step, 2,000 steps, TRL with vLLM rollouts), plus a length control (small length-shaping rewards and a 320-token cap, 384 on Reddit-Books) applied identically to every arm. Evaluation follows Rank-GRPO's protocol with greedy decoding and one list of K=20 "Title (Year)" lines per test request, reporting both per-row metrics (a request counted once per responder) and per-request metrics, with 95% confidence intervals from a paired bootstrap over test requests (2,000 resamples, seed 42). Checkpoints are selected on validation every 250 steps and tested once. Seeds are 3407, then 42 and 1234; several backbones have a single seed.

Why This Matters

Impact on research. The paper isolates a specific, easily overlooked data-construction choice — one training row per annotator — and shows it can dominate the variance of a GRPO advantage estimate and waste most of the RL compute. It connects this to classical Rao–Blackwellization of policy gradients, provides propositions separating binary from fractional rewards under per-group scaling, and reports a transfer result on ProtoQA, which suggests the diagnosis generalizes beyond recommendation to any RL dataset with multiple valid human answers per prompt. The authors also release Reddit-Books and Reddit-Music as IDs and labels.

Real-world applications (as grounded in the paper's setting):

• Forum and community recommenders, such as the Reddit recommendation threads the paper trains on, where a single request receives replies from 5.30 responders on average (Reddit-v2), 4.60 (Reddit-Books) and 7.43 (Reddit-Music).
• Book and music suggestion services, using the Reddit-Books and Reddit-Music pipelines that ground recommendations to the UCSD Goodreads book graph and MusicBrainz artists.
• Question-answering or survey systems with many respondents per prompt, following the ProtoQA setting where about 100 people answer each question.
• General RL post-training pipelines for LLMs that have crowd or multi-annotator data, since the change touches only how training rows are built and scored.

Industry relevance. RL post-training compute is described in the paper as the cost that dominates RL post-training. ReMaRk reaches comparable quality in 25 to 53% of the steps in the reported settings and, in four of seven plateau pairs, reaches Rank-G

Authors’ abstract

A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.