Labels have Human Values: Value Calibration of Subjective Tasks

AI summary

Overview

Research area: Natural Language Processing, specifically subjective/pluralistic NLP, annotation disagreement modeling, and probabilistic calibration of machine learning models.

Technical level: Advanced. The paper assumes familiarity with calibration, ordinal regression, RLHF preference modeling, and transformer encoders.

Scope: The paper introduces MC-STL (MultiCalibrated Subjective Task Learning), a framework that discovers latent human value groups behind subjective annotations and trains models whose predicted probabilities are calibrated within each value group rather than only on average.

What This Paper Is About

Most NLP datasets for subjective tasks (toxicity, safety, preference judgments) are built from multiple annotators who legitimately disagree because they hold different underlying human values. Standard pipelines erase this structure by majority voting or averaging, so a trained model produces probabilities that are systematically wrong for minority value perspectives even when its overall accuracy looks fine. This paper asks whether latent value groups can be recovered from annotation metadata (rationales, expert taxonomies, or annotator sociocultural attributes) and whether a model can be forced to stay calibrated across all of those groups at once.

Key Contributions

1. A value-group construction step with three interchangeable strategies. MC-STL forms value groups either by K-means clustering over sentence embeddings of free-text label rationales, by mapping annotations onto expert-defined value taxonomies (e.g., Schwartz's ten universal value classes, or Helpfulness/Harmlessness in RLHF), or by clustering annotators on sociocultural descriptors such as age, ethnicity, or geographic location, with sociocultural groups acting as proxies for latent value perspectives.

2. A value-aware architecture with value-conditional calibration. Each value group gets a learnable embedding that is added to the text embedding, and annotations that belong to multiple groups receive the average of their group embeddings. Training combines cross-entropy with a KL-divergence term that matches the average predicted distribution inside each value group to that group's empirical label distribution, weighted by a factor alpha_k that up-weights groups whose label distributions diverge from the training set.

3. Coverage of binary, ordinal, and preference-learning settings. Ordinal tasks use the CORAL decomposition into r−1 binary thresholding sub-tasks; the binary setting is the r=2 special case; preference learning uses a Bradley–Terry formulation with a per-value-group learnable temperature, including a multimodal variant where CLIP embeddings of prompt and image are summed.

4. An empirical demonstration of multicalibration and minority-opinion preservation. Across six task configurations spanning value reasoning, toxic chatbot conversations, RLHF preference data, and text-to-image safety, MC-STL is reported to improve overall predictive performance, achieve near-unit calibration slope and near-zero bias within value groups, and score highest on minority opinion alignment.

Main Findings

• Value structure is a usable predictive signal. Grouping annotations by value groups inferred from rationales, expert taxonomies, or sociocultural attributes improved both overall and group-level predictive performance relative to value-agnostic baselines. On VP-Duty (binary, K=3) MC-STL reached 0.78 overall AUC versus 0.69 for Majority Filtering; on VP-Schwartz (binary, K=10) 0.89 versus 0.72; on DICES-990 (ordinal, K=14) 0.76 versus 0.66; on DIVE (ordinal, K=10) 0.68 versus 0.57; on HH-RLHF (preference, K=7) 0.64 pairwise accuracy versus 0.61.

• MC-STL achieves near-ideal value-conditional calibration. Aggregate calibration slopes and biases were close to 1 and 0 respectively on every dataset: VP-Duty 1.00 (0.03) slope, 0.00 (0.03) bias; VP-Rights 0.96 (0.08), 0.00 (0.06); VP-Schwartz 0.98 (0.06), 0.02 (0.06); HH-RLHF 1.00 (0.06), 0.00 (0.03); DICES-990 1.02 (0.04), 0.02 (0.01); DIVE 1.06 (0.07), 0.00 (0.02).

• Value grouping alone is not enough. Value-aware multi-task (VG-MT) and Rating Prediction Module (VG-RPM) baselines had much larger calibration variance. On VP-Schwartz, VG-MT produced a slope of 0.77 (0.08) and bias of 0.13 (0.06), while MC-STL produced 0.98 (0.06) and 0.02 (0.06), indicating that explicitly enforcing group-level distribution matching matters.

• Majority filtering is badly miscalibrated on preference data. On HH-RLHF, Majority Filtering showed a slope of 1.13 (0.15) with bias −0.07 (0.07) and VG-RPM a slope of 1.20 (0.17), which the authors connect to the amplification of overconfident value-class labels when such probabilities are used to filter synthetic preference pairs.

• Annotator-aware modeling beats majority filtering. The Annotator Grouping-MT baseline improved over Majority Filtering on DICES-990 (0.70 versus 0.66) and DIVE (0.58 versus 0.57), confirming that preserving annotator-specific information helps relative to collapsing to majority labels.

• Minority viewpoints are better preserved. MC-STL achieved the strongest minority opinion alignment (1 − normalized EMD) on nearly all datasets: 0.64 (VP-Duty), 0.64 (VP-Rights), 0.71 (VP-Schwartz), 0.62 (HH-RLHF), 0.68 (DICES-990), 0.61 (DIVE), despite minority labels not being a separate training objective.

• Calibration transfers across grouping strategies when value definitions intersect. Training on rationale-clustered groups and evaluating on Schwartz-class groups calibrated the sufficiently represented Schwartz groups near slope 1 and bias 0, while barely represented groups did not. Training on Schwartz groups and evaluating on rationale clusters gave near-ideal calibration across all three rationale groups. The Stimulation class could not be assessed because its annotations (under 0.1 percent) all shared the same label.

• Human verification of the Schwartz mapping. ValuePrism annotations were mapped to Schwartz classes with GPT-4o-mini using a calibrated attributor prompt; a random 3k-sample subset per value class was verified by psychologists with 95 percent agreement.

• Interpretable value group summaries. With K=3 on VP-Duty, the rationale clusters were summarized as a care-centered duty ethic; a protective, law-abiding, harm-averse ethic; and an ethic centered on human dignity, autonomy, truth, fairness, and responsibility.

Methodology in Plain English

The authors start from a dataset where each text item carries several annotations, possibly with free-text rationales, and possibly with annotator demographic metadata.

Step one — build value groups. Depending on what is available, they either (a) embed each rationale with sentence embeddings and run K-means to find clusters of semantically similar value themes, (b) map annotations onto an existing value taxonomy such as Schwartz's ten classes, or (c) group annotations by annotator sociocultural attributes. In the rationale case, they tried K from 2 to 6 and had a domain psychologist pick the final K based on interpretability and uniqueness; other datasets used predefined or fixed K (4, 7, 10, 10, 14 depending on the dataset). Clusters are summarized in natural language, in the rationale case using GPT-4o.

Step two — encode the text and the value perspective separately. A pretrained transformer encoder turns the text into a d-dimensional vector. Separately, each value group has its own learnable d-dimensional vector. For a given annotation, its value vector is the average of the vectors of all groups it belongs to (this handles mixed membership). The two are added together to form the input to a prediction network.

Step three — predict and calibrate simultaneously. The prediction head is either a single sigmoid (binary), a set of CORAL threshold sigmoids (ordinal), or a Bradley–Terry preference score with a per-value-group temperature (preference). The training loss is cross-entropy plus a term that forces the model's average predicted probability inside each value group to match that group's actual label frequency, measured with KL divergence, plus an L2 penalty on the value embeddings. Because the KL term needs hard labels on the predicted side, they use a tanh relaxation with a large constant (l = 10⁴) to allow smooth gradients. Two hyperparameters, lambda_1 and lambda_2, are set by grid search. A weighting factor alpha_k increases the calibration pressure on value groups whose label distribution differs most from the training set.

Step four — evaluate across perspectives. They report overall predictive performance (micro AUC, or pairwise accuracy for preferences), the same metrics computed per value group and averaged, calibration slope and bias computed separately inside each value group using decile binning and a linear fit, and minority opinion alignment measured by normalized 1D Earth Mover Distance to non-majority labels. Experiments run on a single NVIDIA RTX A6000 GPU, with a 20 percent held-out test split built by stratified sampling, two fine-tuning epochs, Sentence-BERT (all-MiniLM-L6-v2) for text embeddings, Longformer-base-4096 for long preference contexts, and CLIP for the text-to-image safety setting.

Why This Matters

Impact on research. The paper reframes annotator disagreement as a signal about latent value structure rather than noise, and connects subjective NLP to the multicalibration literature. It argues that group-level calibration cannot be achieved simply by giving each value group its own prediction head; the distribution-matching term is the ingredient that produces stable calibration. It also offers a diagnostic view of RLHF data, where majority-filtered preference probabilities were observed to be overconfident in a value-dependent way.

Real-world applications.

• Content moderation and toxic chatbot safety, where the same message may be judged differently by different communities and a single averaged toxicity score misrepresents all of them.
• Text-to-image safety filters, where prompt-image pairs are rated on a 5-point safety scale by demographically diverse annotators.
• Preference data curation for RLHF, where value-miscalibrated confidence scores are used to filter synthetic preference pairs and can amplify biases.
• Pluralistic value alignment in value-reasoning systems, where answers about rights, duty, and moral support should reflect normative framing rather than a single dominant viewpoint.

Industry relevance. Any organization deploying models in socially sensitive settings can apply the framework using metadata it may already collect (rater rationales, annotator demographics, or an existing value taxonomy), and the loss is a drop-in addition to standard cross-entropy training. Because K is a configurable number of groups and the method handles binary, ordinal, and preference heads, it maps onto common production setups for safety classifiers and reward models.

Future Directions

• Reduce dependence on value-group quality. The limitations section states that MC-STL assumes subjectivity can be explained through the chosen latent value structure and that its effectiveness depends on the quality of the value groups; the provided text is truncated at this point, so the remaining limitation details are not reported.
• Automate value group count selection. For rationale clustering, K was chosen by a domain psychologist inspecting cluster summaries; replacing this with a principled or automated criterion would make the pipeline cheaper to apply to new tasks.
• Generalize across grouping strategies. The transfer experiment worked for well-represented value groups but not for barely represented ones, such as the Schwartz Stimulation class with under 0.1 percent of annotations, leaving open how to calibrate value groups with very few labels.
• Scale to unseen populations and new modalities. The related work notes that annotator-centric methods struggle to generalize to unseen users, and MC-STL was demonstrated on text and prompt-image pairs, so extending to other modality combinations and to unseen rater populations remains open.

Target Audience

Researchers and practitioners working on subjective NLP, annotation disagreement, pluralistic alignment, reward modeling for RLHF, and model calibration will get the most from this paper. It is also relevant to dataset curators and trust-and-safety teams who collect multi-annotator labels and need models whose confidence is trustworthy across demographic and value perspectives, and to readers interested in how human value theory (Schwartz's taxonomy) can be operationalized inside a training objective.

Authors’ abstract

Although pluralistic societies exhibit diverse human values that lead to legitimate disagreements in subjective tasks (e.g., safety and preference judgments), NLP models trained on such subjective labels often ignore latent value structures, resulting in miscalibrated predictions over relevant value classes. We propose MultiCalibrated Subjective Task Learning (MC-STL), a framework that identifies latent value groups from annotations (via label rationale similarity, expert value taxonomies, or annotator sociocultural descriptors) and enforces value-conditional calibration through value group-specific representations. MC-STL applies to binary, ordinal, and preference learning settings, and is evaluated on multiple datasets covering toxic chatbot conversations, value reasoning, T2I-safety and preference alignment. The results demonstrate that MC-STL consistently outperforms existing baselines, achieving multicalibration across relevant value groups, while delivering gains in probabilistic prediction performance.