Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models

AI summary

Overview

• Research area: AI alignment and safety, specifically the evaluation of language model value preferences (stated vs. revealed preferences).
• Technical level: Intermediate.
• Scope: A systematic study of how the design of preference elicitation protocols—specifically whether models may express neutrality or abstention—changes the measured Spearman rank correlation between stated and revealed preferences across 24 language models.

What This Paper Is About

Language models say they value certain principles, but the choices they make in concrete situations often do not match those stated values. This paper argues that existing measurements of this "stated–revealed (SvR) preference gap" are contaminated by the near-universal use of forced binary-choice prompting, which squeezes indifference, uncertainty, and preference strength into a single forced answer. The authors vary the elicitation protocol systematically—allowing neutrality and abstention at different stages—to see how much the measured gap is a property of the models versus a property of the measurement method.

Key Contributions

1. A systematic comparison of three elicitation configurations (forced–forced, expanded-stated/forced-revealed, and expanded–expanded) across 24 language models, showing that measured SvR correlation is highly protocol-dependent.
2. Quantification of neutrality rates under expanded-choice elicitation for both stated and revealed preferences, revealing wide variation across model families (from 48.2% to 100% for stated preferences).
3. Demonstration that allowing neutrality during stated preference elicitation substantially increases SvR correlation, while allowing it during revealed preference elicitation drives correlation to near-zero or negative values.
4. A test of system prompt steering—conditioning revealed preferences on a model's own stated value ranking over 16 values—showing that this intervention is inconsistent and often detrimental.

Main Findings

• Neutrality is widespread and family-dependent: Under expanded-choice elicitation, stated-preference neutrality rates range from 48.2% to 100% across the 24 models. Qwen-3-8B predominantly selected the "Depends" option; Gemma-3-4B selected "Equal Preference" in approximately 70% of cases; Mistral-3-8B variants selected neutral responses in nearly all revealed scenarios, preventing construction of complete binary rankings, while LLaMA-3.1 and LLaMA-4 largely retained binary decisions.

• Expanded-choice stated preferences raise correlation: Replacing forced-choice stated elicitation with expanded-choice elicitation (while keeping forced-choice revealed preferences) produces a marked increase in SvR correlation. LLaMA-3.1-405B-Instruct improves from ρ ≈ 0.2 to ρ ≈ 0.7. The authors interpret this as neutrality filtering out weak or indeterminate comparisons, leaving rankings that better reflect robust value hierarchies.

• Positive association with capability emerges under that protocol: Under expanded-stated/forced-revealed elicitation, SvR correlation is positively associated with the Epoch Capabilities Index (n=16; Spearman ρ=0.58, p=0.02; Pearson r=0.42, p=0.11). The baseline forced-forced condition shows high variance with no such association (n=16; Spearman ρ=-0.2, p=0.47; Pearson r=-0.23, p=0.38), and the expanded-expanded condition shows low or negative correlation (n=16; Spearman ρ=-0.04, p=0.88; Pearson r=0.1, p=0.7).

• Neutrality in revealed preferences destroys the signal: When both stated and revealed elicitation permit expanded choices, many models consistently choose "Depends" or "Equal Preference" in both conditions, so revealed preferences no longer induce a dense or stable ranking over values, and residual binary choices provide only a weak signal for rank correlation. In Figure 2, models with a neutrality rate above 99% are excluded.

• System prompt steering does not reliably close the gap: Injecting a model's own expanded-choice stated ranking into the system prompt during revealed preference elicitation yields inconsistent effects. A small subset of models (for example, Ministral-3B and Gemma-3-4B) show modest improvements, but many models show reduced SvR correlation, and models from the Claude family consistently regress. The authors connect this to Liu et al. (2025), who report alignment improvements of approximately 23% on HHH-style sets (Askell et al., 2021; 3 values) but only approximately 4% on ModelSpec-style sets (OpenAI, 2025; 6 values); this paper extends the trend to 16 values.

• Why indeterminate responses were excluded: Following survey methodology standards (Krosnick, 1991), neutral responses were excluded from both stated and revealed rankings to isolate strict directional priorities, because retaining them introduces widespread ties that destroy the dense rankings required for SvR correlation. The authors state this methodological choice has consequences for the results, and recommend that future SvR measurement explicitly model neutrality and indeterminacy rather than discard it.

Methodology in Plain English

The researchers built on the LitmusValues framework and AIRiskDilemmas dataset from Chiu et al. (2025), which provides a taxonomy of 16 "Shared AI Values" consolidated from Anthropic's Claude Constitution and OpenAI's Model Spec, including ethical concepts such as Truthfulness, Privacy, Respect, Justice, and Protection, alongside operational values such as Adaptability and Creativity.

Eliciting stated preferences: Models were shown abstract pairwise comparisons between two values, with all 16 values pitted against each other pairwise. Five symmetric prompt templates were used per value pair (differing only in surface phrasing) to control for framing effects. In the expanded-choice condition, the templates explicitly offered four options: prioritize value 1, prioritize value 2, weight them equally, or "it depends / cannot decide."

Eliciting revealed preferences: Models faced 3,000 contextualized moral dilemmas. In the expanded-choice condition, an instruction block allowed "Equal" and "Depends" responses in addition to the two actions, and models had to state their choice (A, B, C, or D) at the start of the response.

Ranking and correlation: Stated rankings were built from win rates over decisive binary comparisons; revealed rankings used Elo ratings converted to a 1–16 scale, derived from "wins" and "losses" across the 3,000 dilemmas. Neutral responses were excluded from both to isolate strict directional priorities. SvR correlation is Spearman's rank correlation between the two 1–16 rankings. All generations used deterministic decoding (temperature=0, top_p=0.01), and responses were categorized into the four response types using an LM judge (GPT-4o-mini).

Steering test: For each model, the authors derived an ordinal ranking over the 16 values from its expanded-choice stated responses, embedded that ranking into a system prompt with strict conflict-resolution instructions, and compared SvR correlation before and after steering on AIRiskDilemmas.

Why This Matters

Impact on research: The paper shows that headline numbers for the stated–revealed preference gap are not stable properties of models but artifacts of elicitation design. It argues that rank-based SvR correlation computed from residual binary choices is an unreliable summary of model behavior when neutrality rates are high, and that measurement should model neutrality and indeterminacy explicitly.

Real-world applications:

• Alignment evaluation and model auditing, where the choice of elicitation protocol can change whether a model appears well-aligned or incoherent.
• Model specification and constitution design, since high behavioral disagreement across value-tradeoff scenarios is informative about where specifications lack granularity or contain contradictions.
• Deployment of agentic systems, where understanding whether a model holds a coherent value hierarchy affects predictions of behavior in high-stakes tradeoffs.
• Preference elicitation in human-facing products, where forced binary prompts may misrepresent model indifference or contextual dependence.

Industry relevance: The paper is directly relevant to teams that build or certify AI systems against published value specifications, and to those using prompt-based steering to shape model behavior. It suggests that simply injecting a stated value hierarchy into the context window is often insufficient to override existing behavioral priors, and may introduce interference that degrades decision consistency.

Future Directions

• Developing measurement methods that explicitly model neutrality and indeterminacy as first-class outcomes rather than discarding them.
• Testing stronger interventions than stated-value system prompting, given that prompt-based steering degrades as the number of values grows; the authors note exploratory work on probe-based steering interventions in the acknowledgements.
• Investigating why steering effects vary so strongly across model families, including the consistent regression observed in the Claude family.
• Understanding what drives the wide variation in neutrality rates across model families and elicitation modes, and whether it reflects genuine contextual sensitivity or training artifacts.
• Determining what SvR correlation means when a model does not express a clean total ordering over values in contextualized scenarios.

Target Audience

AI alignment and safety researchers studying model values and propensities; evaluation engineers designing preference elicitation benchmarks; policy and governance staff who rely on alignment evaluations of model constitutions; and machine learning practitioners applying prompt-based steering to shape model behavior in deployed systems.

Data and code: The paper states that elicitations are available at the Hugging Face dataset LydiaNottingham/MindTheGap and code at https://github.com/SPAR-SvR/Mind-the-Gap. The project was hosted by the Supervised Program for Alignment Research (SPAR), which provided compute resources.

Authors’ abstract

Recent work identifies a stated-revealed (SvR) preference gap in language models (LMs): a mismatch between the values models endorse and the choices they make in context. Existing evaluations rely heavily on binary forced-choice prompting, which entangles genuine preferences with artifacts of the elicitation protocol. We systematically study how elicitation protocols affect SvR correlation across 24 LMs. Allowing neutrality and abstention during stated preference elicitation allows us to exclude weak signals, substantially improving Spearman's rank correlation ($ρ$) between volunteered stated preferences and forced-choice revealed preferences. However, further allowing abstention in revealed preferences drives $ρ$ to near-zero or negative values due to high neutrality rates. Finally, we find that system prompt steering using stated preferences during revealed preference elicitation does not reliably improve SvR correlation on AIRiskDilemmas. Together, our results show that SvR correlation is highly protocol-dependent and that preference elicitation requires methods that account for indeterminate preferences.