What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

AI summary

Overview

Research area: Natural Language Processing / AI alignment, specifically the analysis of human preference data used for finetuning language models.

Technical level: Intermediate. The paper assumes familiarity with preference-based finetuning, reward models, and the basic idea of sparse autoencoders, though the core conceptual contribution is explainable without deep math.

Scope: This paper introduces WIMHF, an automated method for extracting interpretable, human-readable descriptions of what preference datasets encode, and demonstrates its use for auditing, curating, and personalizing preference data across seven widely used datasets.

What This Paper Is About

When language models are aligned using human preference data, the feedback can shape model behavior in unexpected or harmful ways, but practitioners have no systematic way to know what that data actually encodes. Existing approaches either use black-box reward models that predict choices without explaining them, or manually pre-specify features like length or sycophancy, which limits discovery to hypotheses researchers already have. WIMHF solves this by automatically learning interpretable features from the data itself, then measuring which of those features actually drive annotator choices.

Key Contributions

1. A three-step interpretability pipeline for preference data. WIMHF trains a sparse autoencoder on the difference between paired response embeddings, uses an LLM to describe each learned feature in natural language, and fits a logistic regression to identify which features predict the preference label, controlling for length and other confounds.

2. A distinction between measurable and expressed preferences. The paper separates what a dataset can measure (features that vary between the two candidate responses) from what annotators actually express (features that predict the label). This lets practitioners diagnose dataset construction problems before collecting expensive labels.

3. Large-scale empirical analysis across seven feedback datasets. The method surfaces dataset-specific preferences, conflicting preferences between datasets, preferences vulnerable to reward hacking, and unsafe annotations on LMArena that favor toxic content over refusals.

4. Two downstream applications demonstrating utility. Label flipping on LMArena examples that activate an anti-refusal feature raises RewardBench2 safety by 37 percentage points with no loss to other capabilities, and per-annotator weights on subjective style features improve heldout preference prediction for individual users.

Main Findings

• Sparse features retain most of the predictive signal. A logistic regression on just four active SAE features per example achieves an AUC of 0.672 on average, versus 0.766 for a finetuned Llama-3.2-3B reward model. That is 67 percent of the black-box model's improvement over random chance, and 84 percent of what the dense embeddings achieve before sparsification.

• Features align with human explanations and expert judgment. On Community Alignment, 60.4 percent of annotator-written explanations match at least one of the four active SAE features, versus 33.3 percent for a random set of inactive features (p < 0.001). Of 47 features that significantly predict preference across five datasets, external ML researchers rated 41 as helpful and all 47 as interpretable.

• Datasets encode conflicting preferences. Reddit and LMArena users prefer informality, jokes, and flippant tone, while annotators in HH-RLHF, PRISM, and Community Alignment disprefer them. The paper argues this undermines the common practice of mixing datasets for preference finetuning, since opposing signals may cancel out or produce unpredictable model behavior.

• Response sampling strategy shapes what a dataset can measure. PRISM, which samples from 21 different LLMs at high temperature, produces features about style, tone, and refusal. Community Alignment, which prompts a single LLM for four value-diverse candidates, produces features about topic and values diversity with relatively consistent style.

• LMArena annotations encode unsafe preferences. The single largest effect in LMArena is a 31 percent win-rate penalty for refusing user requests. Three of the five highest-impact features are potentially unsafe, including a dispreference for avoiding sexual descriptions. This persists after filtering examples where either response explicitly refused.

• Data curation yields large safety gains. Flipping the chosen and rejected labels for the 1,000 examples with the strongest anti-refusal feature raises Llama-3.2-3B reward model accuracy on RewardBench2's safety subset from 8.9 percent (below the 25 percent random baseline) to 46.2 percent, without degrading math or instruction-following performance.

• Preference data also affects evaluation rankings. Applying the same label-flipping intervention before computing Elo scores shifts 16 of 30 models by at least 50 Elo. Claude-3.5-Sonnet gains 112 Elo and surpasses Gemini-1.5-Pro, while Llama-4-Maverick drops five ranks.

• Style preferences are systematically subjective. Using a random-slopes mixed-effects model, the paper finds that the preference for paragraphs over bulleted lists has by far the largest between-annotator variance (τ = 0.42, compared to 0.22 for the second-most subjective feature). Roughly 18 percent of annotators prefer paragraphs, against a strongly negative overall mean.

• Selective personalization is data-efficient. Learning per-annotator weights only for controllable style features, using 16 examples per annotator and active sampling of high-feature-value examples, improves heldout AUC by up to 1.1 percent over a global model—while deliberately avoiding personalization on political or value-laden axes.

• WIMHF outperforms a prompting-based baseline. Compared to Inverse Constitutional AI, a prompting-based method for the same goal, WIMHF produces more than 1.5 times as many statistically significant preferences and catches important features that the baseline misses, including the misaligned LMArena preferences.

Methodology in Plain English

WIMHF works in three stages.

Stage one: learn what varies between responses. For each preference pair, the method computes text embeddings for the two candidate responses and takes their difference. This difference vector captures how the two responses differ semantically, but it is not human-readable. A sparse autoencoder is trained on these difference vectors to reconstruct them using a small number of active dimensions. The key assumption is that any given response pair differs along only a few interpretable axes out of many possible ones—so the autoencoder is constrained to activate roughly four features per example out of a 32-dimensional latent space. The authors train a separate autoencoder per dataset, so each dataset gets its own vocabulary of features.

Stage two: name each feature. For every learned feature, the method pulls the five preference pairs that most strongly activate it and asks a language model to describe what distinguishes the two responses in those examples. To check that these descriptions are accurate, the method computes a fidelity score: an LLM annotator independently labels whether a feature is more present in one response or the other across 300 held-out examples where the feature is active, and the researchers keep only features whose labels correlate significantly with the autoencoder's activations.

Stage three: find what predicts the label. A logistic regression predicts the preference label from the active features, with word-count difference included as a control so that the analysis does not just rediscover the well-known preference for longer answers. The regression coefficients indicate how much each feature shifts the log-odds of being chosen, and the paper reports an average marginal effect called delta win-rate for readability.

To compare preferences across datasets—which each have their own autoencoders and therefore incomparable feature indices—the authors select a subset of general features and use an LLM judge to annotate 10,000 random examples per dataset for each feature, then compute the same win-rate effect.

Why This Matters

Impact on research. The paper offers a middle path between two extremes in preference-data analysis: fully opaque reward models that predict without explaining, and manually specified feature lists that constrain discovery to pre-existing hypotheses. It also introduces the measurable-versus-expressed distinction, which gives dataset designers a way to audit response diversity before spending money on annotation. The finding that datasets encode directly contradictory preferences has direct implications for how the field should think about dataset mixing.

Real-world applications.

• Dataset auditing before annotation. Practitioners can check whether their sampled responses actually vary along the axes they care about, avoiding the cost of collecting labels that cannot measure the intended preferences.
• Safety-focused data curation. Automatically flagging and re-labeling misaligned examples in preference datasets that will be used for finetuning, as demonstrated with the anti-refusal feature on LMArena.
• Correcting benchmark leaderboards. Recomputing Elo rankings after removing misaligned annotations, which the paper shows can shift model rankings substantially.
• Controllable personalization. Allowing users to specify preferences over response style—such as paragraphs versus bullet points—without opening the door to ideological filter bubbles or personalizing on political content.

Industry relevance. Any organization that collects or purchases human preference data for post-training has a direct interest in knowing what that data encodes. Reward hacking, sycophancy, and overconfidence are known failure modes of preference finetuning, and WIMHF gives teams a concrete tool to anticipate them before training. The method is also cheap relative to finetuning: it operates on embeddings and logistic regressions, not gradient updates on large models.

Future Directions

• Incorporating the prompt into the feature representation. The authors found that using embeddings of the full prompt-response transcript did not improve preference prediction, hypothesizing that prompts are often implied by the responses. They leave a more sophisticated treatment of prompts to future work.

• Extending personalization to larger and more diverse annotator pools. The current personalization results rest on Community Alignment, where annotator IDs and enough per-annotator ratings are available. Scaling this to production settings with sparse per-user data remains open.

• Handling conflicting preferences during dataset mixing. The paper identifies that datasets disagree, but does not propose a principled method for resolving or accounting for those disagreements during training. Standard RLHF does not explicitly model annotator disagreement, so this is a natural next step.

• Generalizing beyond subjective conversations. The analysis deliberately filters out prompts with objectively correct answers, since response correctness likely dominates labels there and text embeddings are unlikely to encode it. Extending interpretability to objective domains would require different representations.

Target Audience

This paper is most useful for AI alignment researchers, post-training engineers at model labs, and data curation specialists who work with human preference data. It also speaks to the growing community studying dataset design and evaluation methodology, and to researchers in computational social science interested in measuring annotator disagreement. A reader needs some familiarity with reward modeling, RLHF, and preference finetuning to follow the experimental setup, but the conceptual framing—learning what feedback data encodes, rather than assuming it encodes what practitioners intend—is accessible to anyone working on model alignment.

Authors’ abstract

Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.