Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

AI summary

Overview

Research area: Natural Language Processing, specifically reward model evaluation and LLM alignment (RLHF).

Technical level: Intermediate. The paper assumes familiarity with reward models, preference data, pairwise ranking loss, PPO-based alignment, and the idea of probing hidden representations.

Scope: The paper proposes and validates a multi-dimensional, probing-based way to evaluate what preference dimensions a reward model actually encodes, rather than reducing evaluation to a single pairwise ranking accuracy.

What This Paper Is About

Reward models are usually evaluated by checking how often they rank a preferred response above a rejected one on a fixed pairwise test set, which collapses evaluation into a binary decision and says nothing about which preference dimensions (harmlessness, correctness, verbosity, and so on) the model has actually learned. The authors instead probe the reward model's internal "preference representation" — the vector at the end token that is mapped to a scalar reward — and train a lightweight classifier on top of it for six separate preference dimensions. They build a benchmark for this (MRMBench) and add an analysis method (inference-time probing) that reveals which dimensions a given reward prediction relies on.

Key Contributions

1. The paper states it is the first work to evaluate whether reward models effectively capture preferences across different dimensions by probing preference representations.
2. It constructs MRMBench (Multi-dimensional Reward Model Benchmark), covering six probing tasks for the dimensions harmlessness, helpfulness, correctness, coherence, complexity, and verbosity, each in an Easy and a Hard version.
3. It introduces inference-time probing, an analysis method that clusters preference representations into per-dimension centroids and uses the distance to those centroids to interpret — and to filter — reward predictions without extra training of the reward model.
4. Through extensive experiments it reports that MRMBench correlates strongly with downstream alignment performance, and that the inference-time probing method improves LLM alignment, reported as an improvement of +5.2 win rate points over baseline on the AlpacaEval test set.

Main Findings

• Reward models do encode human preferences: On MRMBench-Easy, the LLaMA-3.1-8B-Instruct baseline reaches an average accuracy of 75.2%, while a reward model trained on large-scale preference data from the same base model, GPM-LLaMA-3.1-8B, reaches 79.6%. The highest average in the Easy table is GPM-LLaMA-3.1-8B at 79.6%, followed by Eurus-RM-7B at 78.4%.
• Subtle preferences are harder: GPM-LLaMA-3.1-8B scores 79.6% on MRMBench-Easy but 64.5% on MRMBench-Hard. RM-Mistral-7B drops from 76.2% (Easy) to 50.2% (Hard), and BTRM-Qwen2-7b-0613 from 71.3% to 42.9%.
• Harmlessness and coherence degrade less: The paper reports that these two dimensions do not show significant performance degradation between Easy and Hard, which the authors attribute to open-source reward models already being effective at modeling these dimensions even at a subtle level.
• Hard can reorder models: In the Harmlessness dimension of MRMBench-Easy, FsfairX-LLaMA-3-RM-v0.1 outperforms GRM-LLaMA-3-8B-rewardmodel-ft, but under MRMBench-Hard GRM-LLaMA-3-8B-rewardmodel-ft performs better — which the authors read as a sign that FsfairX-LLaMA-3-RM-v0.1 may generalize better on subtle preferences.
• No model wins on all dimensions at once: No reward model ranks high on all six dimensions simultaneously, which the authors attribute to preference training data focusing on certain dimensions and to current optimization methods struggling to balance multiple preference dimensions.
• Harmlessness is consistently strong: Across both MRMBench-Easy and MRMBench-Hard, reward models show robust performance in the harmlessness dimension.
• Strong correlation with alignment quality: Using ten reward models trained with {50k, 100k, 200k, 300k, 400k} preference samples from Unified-Feedback and two LLMs (LLaMA-3.1-8B-Instruct and LLaMA-3.2-3B-Instruct), the Pearson correlation coefficients between MRMBench-Hard scores and aligned-LLM win rates are all greater than 0.8 with p-values smaller than 0.05. Harmlessness win rates use the XStest test set; the other dimensions use AlpacaEval2, with LLaMA-3.1-8B-SFT responses as the baseline.
• Inference-time probing is interpretable and intuitive: For example, the input "How to make a pipe bomb?" tends to have smaller distances to the harmlessness centroids, and pairs closer to the helpfulness centroid tend to provide a more informative response. Some input-response pairs sit far from the centroids of all six dimensions, suggesting the reward model did not rely on these known dimensions, with the underlying mechanism remaining uncertain.
• Probing-derived confidence helps PPO: Using the minimum distance to any cluster centroid as a confidence filter (threshold d_τ = 140), the inference-time probing method reaches a 62.5% win rate versus 57.3% for the Vanilla PPO baseline and 54.3% for the Random-discard baseline.

Methodology in Plain English

A reward model takes an input and a response, reads the representation at the end token of a Transformer decoder, and maps that vector through a linear layer to a single reward score. The authors treat that vector as a "preference representation" and freeze the reward model entirely. For each of six preference dimensions they take an existing annotated dataset, reshape its labels into a balanced Easy version (two classes) and Hard version (three classes), and train only a small linear classifier on the frozen representation to predict the dimension label. Accuracy on that classifier shows how much information about that dimension lives inside the reward model's representation.

For the analysis side, they take the validation set for each dimension, cluster its preference representations into k clusters with K-means (one per label), and treat the resulting centroids as prototypes of that preference dimension. At prediction time, they measure the Euclidean distance from a new input-response pair's representation to each dimension's centroids: a smaller distance means the reward prediction relied more on that dimension. They then reuse this distance as a confidence score during PPO — if a sample's minimum distance to any centroid exceeds the threshold, the reward prediction is treated as unreliable and that sample is dropped from the update.

Label merging was done deliberately for two reasons stated in the paper: to create different evaluation objectives (simple binary probe versus a more nuanced three-way probe) and to fix class imbalance (for helpfulness, only 8% of samples have score 0 while 42% have score 4; merging yields roughly 42% versus 58%).

Why This Matters

Impact on research: Reward model evaluation has largely been a single-number, pairwise-ranking exercise. MRMBench turns it into a per-dimension diagnostic, and the paper reports that these per-dimension scores track downstream alignment quality (Pearson > 0.8, p < 0.05), giving researchers a cheaper alternative to running full alignment experiments. It also supplies evidence that multi-objective optimization in reward modeling is a real open problem, since no evaluated model handles all six dimensions well.

Real-world applications:

• Choosing which open reward model to use for a given application, based on dimension-level scores instead of one aggregate number.
• Auditing safety behavior, since harmlessness is the one dimension where nearly all evaluated models perform well.
• Filtering unreliable reward signals during training to avoid over-optimizing on noisy or out-of-distribution samples.
• Explaining individual reward decisions by showing which preference dimension a given prediction is closest to.

Industry relevance: Companies that fine-tune LLMs with RLHF or use reward models for Best-of-n reranking need to know whether their reward model actually captures the behaviors they care about. This work offers a reusable benchmark plus a training-time filter that the authors report improves win rate without retraining the reward model, which is attractive for teams that already have reward models in production.

Future Directions

• Expanding beyond the six dimensions: the paper notes that harmlessness itself could be subdivided by culture or values (for example religious-related harmlessness, or harmlessness in Western versus Eastern culture), and demonstrates guidelines for adding dimensions using fairness and ethics as case studies.
• Investigating the input-response pairs that are far from every centroid, since the reward model's mechanism for those cases remains uncertain.
• Testing the dynamic, distance-thresholded PPO filtering across more thresholds, seeds, and reward models (the paper notes varying thresholds are covered in its appendix).
• Better training strategies for balancing multiple preference dimensions at once, which the paper points to as the promise of multi-objective optimization in reward modeling.

Target Audience

Researchers and engineers working on LLM alignment, RLHF, and reward modeling who need finer-grained evaluation than pairwise ranking accuracy; practitioners selecting or auditing open-source reward models; and NLP researchers interested in representation probing as an evaluation and interpretability tool.

Authors’ abstract

Previous methods evaluate reward models by testing them on a fixed pairwise ranking test set, but they typically do not provide performance information on each preference dimension. In this work, we address the evaluation challenge of reward models by probing preference representations. To confirm the effectiveness of this evaluation method, we construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions. We design it to favor and encourage reward models that better capture preferences across different dimensions. Furthermore, we introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability. Through extensive experiments, we find that MRMBench strongly correlates with the alignment performance of large language models (LLMs), making it a reliable reference for developing advanced reward models. Our analysis of MRMBench evaluation results reveals that reward models often struggle to capture preferences across multiple dimensions, highlighting the potential of multi-objective optimization in reward modeling. Additionally, our findings show that the proposed inference-time probing method offers a reliable metric for assessing the confidence of reward predictions, which ultimately improves the alignment of LLMs.