P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

AI summary

Overview

• Research area: Natural Language Processing — personalized alignment of large language models (LLMs) and generative reward modeling.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning from human feedback (RLHF), GRPO, DPO, Bradley–Terry reward models, LoRA, K-means clustering, and PCA.
• Scope in one sentence: The paper introduces P-GenRM, a personalized generative reward model that converts hybrid user preference signals into structured, scenario-aware evaluation chains and adds a test-time "user-based scaling" mechanism that aggregates individual-level and prototype-level (similar-user) scoring schemes.

What This Paper Is About

Personalized alignment wants an LLM to match one particular user's taste rather than generic values like helpfulness or harmlessness, which requires a reliable per-user reward signal. Existing personalized reward models have two weaknesses: they flatten varied, situation-dependent preferences into a small fixed set of evaluation rules, and they generalize poorly to new users who have little feedback history. P-GenRM addresses both by generating an adaptive persona and scoring rubric per query, and by borrowing preference signal from similar users at inference time.

Key Contributions

1. First personalized generative reward model with structured evaluation chains. P-GenRM transforms heterogeneous preference signals (implicit interaction history plus explicit stated criteria) into interpretable evaluation chains consisting of a scenario-specific user persona, derived preference criteria, and weighted scoring rubrics in open-domain settings.
2. Test-time User-based Scaling. The paper proposes a dual-granularity scaling mechanism that exploits the test-time scalability of generative reward models: individual-level scaling (parallel sampling of multiple scoring schemes for the same user) and prototype-level scaling (incorporating scores inferred from similar users).
3. A three-stage training framework. Persona-guided Scoring Induction (PSI) via supervised fine-tuning, Criteria-based Reasoning Enhancement (CRE) via reinforcement learning with combined process and outcome rewards, and hard-negative-aware curriculum learning.
4. Strong empirical results and cold-start generalization. State-of-the-art results on personalized reward benchmarks plus a measured boost from test-time scaling, and improved Spearman correlation on an out-of-distribution dataset with sparse user feedback.

Main Findings

• State-of-the-art on PersonalRewardBench: P-GenRM achieves an average improvement of 2.31% over prior methods (abstract). Broken down by scale in the experiments section, the 8B model improves by an average of 2.77% over the prior state of the art, and the 70B LoRA-trained model by an average of 1.99%.
• 8B beats the previous best 70B: P-GenRM-8B surpasses the previously best-performing 70B model by an average of 1.04%.
• Test-time scaling adds roughly 3%: Test-time User-based Scaling provides an additional 3% boost (abstract), described in the analysis as an average improvement of 2.99% over P-GenRM itself for the best setting.
• Best scaling configuration is Ind-16, Pro-8: This setting reaches 75.92 ± 1.70% on Chatbot Arena-Personalized and 68.06 ± 0.69% on PRISM-Personalized, outperforming Ind-32 while using fewer scaling steps (16+8).
• More similar-user ratings is not always better: Increasing prototype-level ratings without limit does not necessarily help — Ind-0, Pro-8 scores 66.90 ± 1.54% (Chatbot Arena) and 57.65 ± 0.86% (PRISM), and Ind-16, Pro-16 drops to 72.59 ± 1.61% / 64.61 ± 0.72%, underscoring the highly user-specific nature of the task.
• P-GenRM-8B beats OpenAI o3 with prompting: P-GenRM (8B) scores 72.68 ± 1.85% / 65.32 ± 0.56% versus o3 at 64.47 ± 1.62% / 56.34 ± 0.64%, and versus o3 + PSI at 69.14 ± 1.46% / 63.87 ± 0.85%.
• Macro accuracy: P-GenRM achieves the highest macro accuracy (65.21%) among all evaluated baselines, indicating it does not overfit to any majority persona.
• Every training stage matters: Ablations on P-GenRM (72.68 ± 1.85% / 65.32 ± 0.56%) degrade to 71.07 ± 1.44% / 63.82 ± 0.64% without curriculum learning, 70.22 ± 1.74% / 62.70 ± 0.73% without CL and process reward, 69.05 ± 1.59% / 60.94 ± 0.77% without CL and outcome reward, 66.76 ± 1.42% / 57.08 ± 0.89% without CL and RL, and 56.37 ± 2.16% / 52.04 ± 0.54% without CL, RL, and SFT.
• Adaptive personas beat static ones: PSI outperforms SynthesizeMe (SMe) across base models — Qwen3-8B goes from 61.82 ± 1.47% / 55.01 ± 0.77% to 64.22 ± 1.58% / 58.01 ± 0.83% with PSI versus 62.57 ± 1.84% / 56.33 ± 0.91% with SMe; o3 goes from 64.47 ± 1.62% / 56.34 ± 0.64% to 69.14 ± 1.46% / 63.87 ± 0.85% with PSI versus 67.73 ± 1.94% / 58.49 ± 1.22% with SMe.
• Prototype count of 50 is the chosen setting: Performance improves substantially as prototypes increase from 0 to 50, plateaus after, and slightly degrades at 100; the authors set the number of prototypes to 50 based on PCA retained-variance analysis.
• Cold-start generalization on LaMP-QA: P-GenRM (8B) with Ind-8, Pro-4 achieves the best average Spearman correlation of 0.638 (Arts 0.543, Personal Life & Development 0.714, Society & Culture 0.657), surpassing all baselines including Qwen3-235B-A22B at 0.619, LLaMA3.1-70B at 0.600, SynthMe-8B at 0.581, Qwen3-32B at 0.562, Qwen3-8B at 0.543, and LLaMA3.1-8B at 0.524.
• Latency: Test-time user-based scaling incurs only a limited increase in inference time while retaining lower latency than prior state-of-the-art methods; the specific measurements are presented in the paper's Appendix A.9.
• Policy training: Training policy models with P-GenRM under DPO and GRPO settings boosts an 8B policy model to surpass the performance of 70B-sized models (details in Appendix A.12).
• Broader preference space: While the PRISM dataset offers predefined criteria such as {Style, Values, Fluency, Factuality, Safety, Diversity, Helpfulness}, P-GenRM uses a wider range of personalized dimensions including {Philosophical Engagement, Openness, Structure, Depth, Nuance, Sensitivity, Breadth of Ideas}.

Methodology in Plain English

The approach has two halves: how the reward model is trained, and how it is used at inference time.

Training (three stages). First, Persona-guided Scoring Induction: an instruct LLM is prompted with both the user's implicit signals (sampled history of preferred and dispreferred responses) and explicit stated criteria, and is made to write out a scenario-specific persona, derive criteria from it, and score candidate responses. The data that survives rejection sampling trains P-GenRM via supervised fine-tuning. Second, Criteria-based Reasoning Enhancement: reinforcement learning built on GRPO teaches the model to produce good evaluation chains even when explicit feedback is missing — explicit criteria are used as training supervision rather than a test-time requirement. The reward combines a process reward from an LLM-as-a-judge (scored 0 to 1, checking whether the reasoning covers the user's stated or synthetic explicit preferences) with a rule-based outcome reward that checks whether the chosen response got a higher score than the rejected one, plus a −0.1 penalty for formatting errors; these are combined with weights α and β (set to 0.5 and 1.0 in the RL stage). Third, hard-negative-aware curriculum learning: the share of difficult negative cases is gradually increased, and the process reward is disabled in this stage to allow a larger exploration space. Training used LLaMA-3.1-8B and LLaMA-3.1-70B (LoRA for 70B), with 8 and 32 GPUs respectively.

User prototypes. Each inferred preference analysis is embedded using Qwen3-Embedding-0.6B; the embeddings form a cross-scenario matrix that is clustered with K-means into k centroids acting as user prototypes. These prototypes are then refined with a history-aware attention mechanism that weights past interaction triples by relevance to the current query, plus a pairwise discrimination loss and two regularizers — one pulling each prototype toward its cluster center, one keeping its evolution smooth across updates.

Inference (dual-granularity scaling). For a query, P-GenRM samples m individual scoring schemes for the user and also finds the user's nearest prototype, selects the n most similar users, and averages in their scores. The final score is the mean over the m individual schemes plus the mean over the n similar-user schemes. The intuition: individual sampling explores multiple hypotheses about a user's taste, while prototype sampling suppresses noise and transfers knowledge to new users with sparse history.

Why This Matters

• Research impact: The work reframes personalized reward modeling as generative, interpretable reasoning over preference signals rather than fitting a fixed latent or a fixed set of evaluation dimensions. It also imports collaborative-filtering ideas (user prototypes, similar-user aggregation) into reward modeling, and shows that test-time scaling — previously demonstrated for generic generative reward models — can be made user-specific.
• Real-world applications:
  • Dialogue assistants that adapt tone and style per user (e.g., the paper's own example: brief while driving, expressive in casual settings).
  • Cold-start personalization in products where a new user has almost no feedback history, handled via prototype-based transfer.
  • Reward modeling for subjective or open-ended generation tasks — creative writing, advice, opinionated Q&A — where evaluation depends on individual standards.
  • Reinforcement learning pipelines for policy training, where P-GenRM can act as the reward signal (demonstrated under DPO and GRPO).
• Industry relevance: The results are delivered at 8B and 70B scales, and the 8B model outperforms the prior best 70B model as well as the proprietary o3 with prompting, which matters for cost-sensitive deployment. The authors are from the Qwen-Character Team at Alibaba Group, and code is released at the linked GitHub repository, signaling direct production intent.

Future Directions

• Balancing scaling budget: The paper shows non-monotonic returns — prototype counts beyond 50 plateau or slightly degrade, and Ind-16/Pro-16 underperforms Ind-16/Pro-8 — leaving open the question of how to adaptively choose m and n per user or per query rather than fixing them.
• Prototype granularity: How to determine the optimal number of user prototypes automatically, and whether hierarchical or fine-grained partitioning can avoid the inference noise observed at 100 prototypes.
• Preference drift over time: The formulation assumes a user's history up to turn t, but the paper does not report results on long-horizon preference shift; extending the model to track changing users is a natural next step.
• Reward-model-to-policy loop: The paper reports that P-GenRM boosts an 8B policy model past 70B-sized models under DPO and GRPO, but details are deferred to the appendix, so the full picture of downstream policy quality and reward hacking remains an open area.

Target Audience

Researchers and engineers working on LLM alignment, RLHF/RLVR, and reward modeling who need per-user rather than universal preference signals; practitioners building personalized assistants or recommendation-adjacent dialogue systems with cold-start constraints; and anyone interested in test-time compute scaling applied to subjective evaluation. Readers should be comfortable with reinforcement learning objectives, clustering, and standard reward-model benchmarks to get full value from the tables and formulas.

Authors’ abstract

Personalized alignment of large language models seeks to adapt responses to individual user preferences, typically via reinforcement learning. A key challenge is obtaining accurate, user-specific reward signals in open-ended scenarios. Existing personalized reward models face two persistent limitations: (1) oversimplifying diverse, scenario-specific preferences into a small, fixed set of evaluation principles, and (2) struggling with generalization to new users with limited feedback. To this end, we propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling. P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics across various scenarios. It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism: at the individual level, it adaptively scales and aggregates each user's scoring scheme; at the prototype level, it incorporates preferences from similar users. This design mitigates noise in inferred preferences and enhances generalization to unseen users through prototype-based transfer. Empirical results show that P-GenRM achieves state-of-the-art results on widely-used personalized reward model benchmarks, with an average improvement of 2.31%, and demonstrates strong generalization on an out-of-distribution dataset. Notably, Test-time User-based scaling provides an additional 3% boost, demonstrating stronger personalized alignment with test-time scalability.