Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences

AI summary

Overview

Research area: AI alignment and evaluation, specifically whether large language models trained on human preference data learn deeper human values or only surface-level patterns.

Technical level: Intermediate. The experimental design and metric are straightforward, but the paper assumes familiarity with in-context learning, reward hacking, and AI alignment vocabulary.

Scope: The paper introduces the Deep Value Benchmark (DVB), a controlled confound-then-deconfound evaluation that measures a model's Deep Value Generalization Rate (DVGR) across 9 language models, 11 deep values, 20 shallow preferences, and 8 application domains.

What This Paper Is About

When LLMs are trained on human preferences, it is unclear whether they internalize the deeper values motivating those choices or merely copy superficial correlations in the data. The authors build an experiment where deep values (moral principles such as justice or non-maleficence) are deliberately correlated with shallow features (such as formal versus informal writing style) during a training phase, then decoupled during a test phase. The goal is to measure whether a model follows the underlying value or the surface feature when the two conflict.

Key Contributions

1. A measurement framework. The "confound-then-deconfound" design creates controlled experiments where correlated deep and shallow attributes are deliberately separated, revealing which signal a model actually generalizes. The authors present this as a general approach extendable beyond values versus preferences.

2. A validated dataset and interpretable metric. The DVB dataset was subject to three separate human validation experiments (covering construct validity, internal validity, and external validity), and is released publicly. DVGR is offered as an interpretable score of deep value generalization, where 1 means always following the deep value and 0 means always following the shallow feature.

3. Empirical results across 9 models. The authors measure DVGR for gemini-2.0-flash-lite, gemini-2.0-flash, gpt-4o-mini-2024-07-18, gpt-4o-2024-08-06, gpt-4.1-nano-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-2025-04-14, llama-3-8b-instruct, and llama-3-70b-instruct, finding that all generalize shallow preferences rather than deep values.

4. Follow-up prompt experiments. Chain-of-thought prompting and explicit instructions to generalize the deep value were tested against a baseline prompt, showing that explicit instruction helps only somewhat while chain-of-thought hurts.

Main Findings

• Average DVGR of 0.30. Across 9 models, the average Deep Value Generalization Rate was 0.30, and for every model DVGR was significantly below chance. The analysis dataset contained N = 104,725 trials, with responses extracted in 97% of trials.

• Model size does not help. Within pairs of smaller and larger models from the same family, smaller models had a higher DVGR in 3 of 5 comparisons. All differences were significant due to large sample size, but effect sizes were small (mean absolute DVGR difference = 0.07). An omnibus chi-squared test also showed smaller models slightly outperforming larger ones.

• Models behave similarly to each other. Across all model pairs, average agreement was 74%. Same-developer pairs agreed 76.8% of the time versus 72.2% for different-developer pairs, a difference of 3.6 percentage points (95% CI [0.4, 6.8], p = 0.04).

• Value type matters more than context or example count. DVGR varied strongly by value (Cramér's V = 0.18), with tradition scoring 0.51 (95% CI [0.50, 0.52]) and universalism 0.42 (95% CI [0.41, 0.43]), while fidelity and self-improvement were lowest. Context effects were weaker (Cramér's V = 0.09), with commerce, healthcare, and finance the highest-DVGR contexts and communication, education, and customer service the lowest. In-context example count had almost no effect (Cramér's V = 0.01): DVGR was 0.31 at n = 4 (95% CI [0.30, 0.31]), 0.30 at n = 20 (95% CI [0.30, 0.30]), and 0.30 at n = 40 (95% CI [0.29, 0.30]).

• Perceived value popularity and distinctiveness predict generalization. Asked to rate values on popularity, distinctiveness, and predictiveness (average SD < 0.5 across repeated ratings), models generalized values they perceived as unpopular (odds decrease 14.44% per unit increase in popularity; OR = 0.86, p < .001) and distinctive (odds increase 24.57% per unit increase; OR = 1.25, p < .001). Perceived predictiveness had no effect. The authors describe this analysis as correlational and exploratory.

• Explicit instruction helps only somewhat; chain-of-thought hurts. Pooling across models, chain-of-thought reasoning produced a DVGR of 0.25 (95% CI [0.24, 0.26]), below the baseline of 0.30 (95% CI [0.30, 0.31]). Explicitly instructing models to generalize the deep value produced 0.33 (95% CI [0.32, 0.34]), still below chance.

• Value confounds do not explain the results. Fitting mixed models with random intercepts for each preferred deep value produced an adjusted DVGR nearly identical to raw estimates (mean absolute difference = 0.003).

Methodology in Plain English

The researchers built paired choices about how an AI agent should behave, where each option combines one deep value with one shallow preference, such as "non-maleficence plus formal language" versus "justice plus informal language." Deep values came from two established frameworks: six of W.D. Ross's prima facie duties (beneficence, fidelity, justice, non-maleficence, reparation, self-improvement) and five of Schwartz's social values (security, conformity, tradition, universalism, benevolence). Shallow preferences were generated by asking GPT-4o for candidate dichotomies over 10 trials, de-duplicating to 38 candidates, then having crowdworkers rate each on shallowness, preference neutrality, and domain breadth. The top 20 were selected by ranking on 0.5 times the rank of neutrality plus 0.5 times the rank of breadth.

Human validation confirmed that people distinguished the two categories: shallowness ratings for deep values (M = -0.98, SD = 1.00, Mdn = -1.00) were lower than for shallow preferences (M = 0.34, SD = 1.36, Mdn = 1.00), a large effect of d = 1. Binarized accuracy was 0.7 on the full dataset and 0.9 for shallow preferences returned by the ranking algorithm.

Contexts were grounded in real applications. The authors recorded tags from a Y Combinator page listing "100 of the top AI Assistant startups" as of April 2025, filtered 83 unique tags down to 40 domain-application tags, clustered them into 11 groups, and kept the 8 clusters with at least five tag appearances: commerce, customer service, finance, productivity, communication, healthcare, legal, and education. Work activities within each cluster were drawn from O*NET, the Department of Labor occupational database, by mapping clusters to Standard Occupational Classification codes and selecting the top 10 most relevant activities per cluster.

From the resulting combinatorial universe, they sampled 50 value-and-preference pairings per context, giving 400 tuples. Each tuple produced 40 training examples and 40 test examples, yielding 32K completions. For each test trial, a model received N in-context training examples where the user always preferred one combination, followed by a single test question with the shallow features swapped. N was 4, 20, or 40, with 10 test questions per tuple, giving 400 × 3 × 10 = 12K test questions, each administered as its own prompt. DVGR is the proportion of trials where the model chose the value-aligned option. Two additional validation studies checked that humans could identify which option embodied which value (200 trials, participants predicted the preferred option in 91% of cases) and that the generated options correctly embodied their assigned value and preference (210 trials, 98% accuracy). Experiments ran on a university high-performance computing cluster using 32 CPU cores over 2 days of CPU time and 4-hour runtime, with default temperature settings and max tokens set to 10.

Why This Matters

The paper's central claim is that preference-trained systems may be learning statistical patterns that correlate with human choices rather than the values behind them, which could cause consequential failures as AI systems gain autonomy. It offers an interpretable metric researchers can track across model generations, and a general experimental template for detecting what signals a model generalizes whenever deeper intent must be separated from surface correlation.

Real-world applications:

• Healthcare assistants. A system that recommends doctors based on a correlated specialty rather than a patient's underlying value of thorough communication could steer users away from better-fitting specialists.
• Legal and financial agents. These are among the contexts with higher DVGR in the study, and they represent high-stakes delegated decisions where shallow generalization could produce misaligned recommendations.
• Consumer-facing agents in commerce and customer service. These appear in the study's context analysis and reflect everyday deployments where users expect agents to act on their intent.
• Model selection and procurement. DVGR gives organizations a way to compare vendors on a property that model size does not predict.

Industry relevance centers on the finding that scale does not reliably improve deep value generalization, and that models from the same developer answer the benchmark more similarly than models from different developers. Given existing market concentration in foundation models, the authors argue this poses a risk for pluralistic AI.

Future Directions

• Fine-tuning experiments. The study tested inference-only, in-context learning with off-the-shelf models. The authors ask whether models can be fine-tuned to generalize the deep value, and what downstream behaviors that would affect.

• Tracking DVGR across model generations. Because the metric is interpretable and the dataset is released, researchers can monitor whether deep value generalization improves over time.

• Understanding why models fail. The authors state that the cause of low DVGRs is unclear. They note the deep value is more abstract than the shallow preference and connect the result to low LLM performance on other abstraction tasks.

• Testing partial, naturalistic confounds. The design deliberately creates a perfect confound between values and preferences, which the authors describe as a worst-case scenario. How models behave when confounds are partial and variable remains open.

Target Audience

AI alignment and safety researchers; evaluation and benchmarking scientists; model developers at labs that train on human preference data; and policy or procurement staff who need an interpretable measure of whether a deployed system generalizes user intent. Readers interested in machine ethics and value theory will also find the operationalization of Ross's prima facie duties and Schwartz's basic values useful.

Authors’ abstract

We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features -- for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR) -- the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.