When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

AI summary

Overview

Research area: Natural Language Processing / speech generation — specifically reinforcement-learning post-training of codec-based text-to-speech (TTS) models using learned perceptual ("subjective") reward predictors, with human listening validation.

Technical level: Advanced. The paper assumes familiarity with codec speech language models, Group Relative Policy Optimization (GRPO), reward overoptimization, and human A/B evaluation methodology.

One-sentence scope: The paper systematically tests when optimizing learned subjective predictors — anime-like speaking style, naturalness, likability, and arousal — with GRPO in a Llasa/XCodec2 TTS model actually moves the model in directions human listeners perceive, rather than only inflating predictor scores.

What This Paper Is About

Codec-based TTS models generate speech as discrete acoustic tokens, which means they can be post-trained with the same reinforcement-learning machinery used for large language models. Unlike transcription accuracy, however, attributes such as naturalness, likability, and anime-like speaking style are defined by listener judgments and can only be approximated by learned predictors — proxies that a policy can exploit without producing speech listeners actually prefer.

This paper asks when a learned perceptual predictor can serve as an RL reward while staying aligned with human listeners. The authors train GRPO on four subjective reward axes under a character-error-rate (CER) constraint, compare it against Best-of-N reranking, and run multi-rater Japanese listening tests to measure whether machine-level predictor gains transfer to human preference.

Key Contributions

1. A controlled training-and-evaluation scaffold for subjective speech rewards. GRPO is instantiated for learned perceptual predictors under a CER-zone hard constraint, compared against Best-of-N reranking to separate reward-based sample selection from policy-level movement, and evaluated in both first-shot and CER-retry modes used for human evaluation.

2. A multi-rater human study of predictor–axis–base tuples. The authors show that machine-level reward gains transfer unevenly to listeners: some tuples produce strong human-aligned shifts, some only modest transfer, and some fail on average despite high-confidence successes.

3. Diagnostics for reward signal quality. Reward-gap calibration, base-output score spread, domain match, and within-zone signal strength are evaluated as diagnostics for deciding which subjective rewards are suitable for RL or future multi-reward post-training.

4. Released artifacts. Code, prompts, generated audio samples, and reward scores are released at https://github.com/sizigi/animeGRPO, with an audio demo at https://sizigi.github.io/animeGRPO/.

Main Findings

• Subjective rewards are not interchangeable quality surrogates. In the cross-axis mean objective shifts (Table 1), the largest positive shift on each row appears on the optimized axis: AnimeScore GRPO shifts AnimeScore by +1.353 with a CER change of −0.030; Likability GRPO shifts Likability by +0.167; UTMOS GRPO shifts UTMOS by +0.485. Every other axis moves much less or negatively on an off-axis metric.

• The CER-zone constraint changes the reward–intelligibility trade-off. On AnimeScore, Zone-CER produces a larger target shift (+1.35) and lower median CER (0.054) than Best-of-8 (+1.21, 0.070), but Best-of-8 has fewer violations (10.0% vs 16.0%). Removing the zone (Target-only) worsens median CER (0.087) and violations (24.0%) relative to Zone-CER.

• GRPO is not uniformly better than reranking. On UTMOS, Best-of-8 (+0.47) and Zone-CER (+0.49) achieve almost identical target gains, with Best-of-8 at lower median CER (0.055 vs 0.074) and Zone-CER at fewer violations (6.0% vs 10.0%). On Likability, Zone-CER gives the largest target gain (+0.17) but with higher median CER (0.098) than Best-of-8 (0.043), while still having fewer violations (6.0% vs 10.0%).

• Human transfer is uneven across axes. In GRPO vs Base item-level comparisons (Table 3), AnimeScore reaches HWR 80.0, MWR 88.0, and agreement 88.0. UTMOS reaches HWR 62.0, MWR 74.0, agreement 80.0. Likability shows HWR 36.0, MWR 56.0, agreement 76.0 — humans prefer the base overall.

• Best-of-8 is a strong human-level baseline. GRPO vs Best-of-8 human preference is near chance: HWR is 52.0 for AnimeScore, 46.0 for UTMOS, and 48.0 for Likability. GRPO should be viewed as amortizing reward-selected behavior into the policy rather than outperforming reranking.

• Signed reward gaps predict listener choices; residual CER gaps do not. In the pooled vote-level logistic regression (Table 4), a one-standard-deviation increase in reward gap in favor of GRPO increases the odds of choosing GRPO by 1.93 times (β = +0.657, SE 0.172, 95% CI [1.38, 2.70], p < .001). The CER gap coefficient is −0.041 (SE 0.186, OR 0.96, 95% CI [0.67, 1.38], p = .83), and a Wald test rejects equality of the two slopes.

• Per-axis calibration is heterogeneous. Likability and UTMOS have significant reward-gap slopes, while AnimeScore slopes are positive but not significant (Appendix F). Likability therefore fails on average yet still aligns in high-confidence regions.

• Base-output spread tracks transfer in part. Base predictor distributions (Table 5) show AnimeScore mean −0.39, std 1.53, range [−2.52, 4.20]; UTMOS mean +3.08, std 1.01, range [1.30, 4.23]; Likability mean +4.20, std 0.44, range [2.78, 4.58]. The widest spread corresponds to the strongest positive transfer, and the narrowest corresponds to the average-failure case.

• Constrained-scaffold failure mode for arousal. In the VAD-Arousal run, validation reward improved mainly by reducing CER violations while validation arousal stayed within seed-level variation. Because no human A/B study was conducted for this axis, the authors treat it as a training-dynamics negative case rather than evidence about perceptual transfer.

• Conclusions are robust to residual CER violations. Excluding residual violator pairs changes vote-level HWR by +1.9 pp for AnimeScore, −2.1 pp for UTMOS, and −4.5 pp for Likability, leaving qualitative conclusions unchanged.

Methodology in Plain English

The backbone is Llasa (a public Llasa-1B-Multilingual checkpoint) with XCodec2 as the acoustic tokenizer and waveform decoder; GRPO is implemented on top of verl with vLLM rollouts, running on one H100 80 GB GPU for the actor and a second H100 80 GB for reward predictors.

Rewards are wrapped in a three-zone template. A Whisper large-v3 transcript of the generated waveform is compared to the input text to compute CER. If CER is at or below 0.10 (CLEAN zone), the normalized predictor score gets a +0.5 bonus; if CER is between 0.10 and 0.30 (FEASIBLE zone), the predictor score is used alone; if CER exceeds 0.30 (VIOLATE zone), the output receives a fixed −1.0 penalty. This hard-zone design prevents a high perceptual score from numerically compensating for a transcription failure, and the same thresholds (τ_l = 0.10, τ_h = 0.30, b = 0.5, ρ = 1.0) are used in all experiments.

Three learned predictors are used in the main experiments: AnimeScore (a pairwise-preference-trained anime-likeness predictor with a WavLM-base encoder and RankNet-style ranking head, normalized as max(0, p(s)+3.0)); UTMOS, used off-the-shelf from UTMOS22-strong (MOS-like score on [1,5], divided by 5); and Likability, trained for this study on CocoNut-Humoresque using a WavLM encoder and a six-class softmax expectation (normalized as (p(s)−1)/5). A fourth off-the-shelf MSP-Dim arousal predictor is used only for a training-dynamics diagnostic.

GRPO samples K = 4 rollouts per prompt, subtracts an adaptive in-reward KL penalty from the zone reward, and normalizes rewards within the rollout group. Checkpoints are not selected by raw predictor score; instead the checkpoint maximizing constraint-aware validation reward is chosen while monitoring CER violations and KL drift. Training uses 900 Japanese Wikipedia-derived prompts, checkpoint selection uses a disjoint 100-prompt validation set, and evaluation uses 50 held-out Japanese prompts partitioned into emotional, anime-stylized, neutral, long-form narrative, and linguistically challenging groups.

Two evaluation modes are distinguished. First-shot evaluation uses a single stochastic generation with a fixed seed and no filtering. CER-retry evaluation (used to prepare audio for human tests) regenerates with retry seeds when CER exceeds 0.30 and takes the first candidate at or below 0.30, or the lowest-CER candidate if none qualifies — applied symmetrically to both sides of every pair, with all 50 prompts retained.

Human evaluation uses pairwise listening tests through the Japanese crowdsourcing platform Lancers, covering three reward axes. Each axis contains 50 paired items, each item receives 5 independent ratings, and each axis is rated by 10 distinct Japanese listeners with no listener overlap between axes. Listeners are blinded to side assignment, and win rates are computed at the item level using majority vote, with Wilson 95% confidence intervals reported in Appendix E.

Why This Matters

Impact on research. The paper supplies a concrete methodological reframing: rather than asking whether subjective rewards "work," it argues that each reward setting should be analyzed as a predictor–axis–base tuple, and it separates three phenomena that are often conflated — whether a predictor can be optimized, whether the resulting policy shift stays human-aligned, and whether the reward is suitable for inclusion in future multi-reward post-training. It also provides a reusable diagnostic set (reward-gap calibration, base-output spread, domain match, within-zone signal strength) and evidence that Best-of-N reranking is a strong baseline that policy optimization must be compared against.

Real-world applications:

• Screening reward predictors before expensive multi-reward speech post-training, using the four proposed checks rather than raw predictor-score gains.
• Improving controllable TTS for creative and media production, including anime-style voice acting performance and stylized narration.
• Building more natural-sounding speech synthesis and voice interfaces where listener-perceived naturalness (UTMOS-style quality) is the target.
• Reducing intelligibility regressions in RL-tuned TTS by enforcing an intelligibility zone so that perceptual gains cannot be bought with transcript drift.

Industry relevance. The findings suggest that inference-time Best-of-N reranking is a competitive and cheaper alternative to policy optimization for some subjective axes, which matters for deployment cost. The diagnostic framework also gives product teams a way to avoid shipping policies that score well on an internal proxy but are not preferred by listeners.

Future Directions

• Disentangle predictor architecture, perceptual axis, and base-model distribution. The authors state their results should be read as diagnostics for predictor–axis–base tuples rather than causal claims that one axis is intrinsically easier or harder, since the reward models differ in training data, score scale, target construct, and base-model coverage.
• Scale the evaluation. Larger studies could add more base models, broader listener populations, native English listeners for the auxiliary English condition, and larger reranking budgets.
• Strengthen the training-side analysis. The limitations section calls for fixed-KL comparisons, multi-seed training, and direct within-prompt rollout-spread measurements.
• Extend to multi-reward post-training. The diagnostics are positioned as a pre-screening step before combining multiple subjective rewards, but the paper does not itself report multi-reward post-training results.
• Investigate fairness and cultural variation. The ethics section calls for examining how style-control rewards behave across demographic groups, listener communities, and culturally specific notions of expressiveness or likability.

Target Audience

This paper is most valuable to speech-generation researchers and engineers working on reinforcement-learning post-training of codec-based TTS, especially those considering learned perceptual predictors as rewards. It is also directly useful to practitioners building reward-model pipelines for audio — evaluation scientists, RLHF/reward-modeling engineers, and product teams deciding between policy optimization and inference-time reranking. Readers who need background in GRPO, codec speech language models, and human A/B evaluation methodology will find the paper advanced; those specifically interested in the human-validation side of speech synthesis evaluation will find the multi-rater protocol and reward-gap calibration analysis the most transferable elements.

Authors’ abstract

Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.