VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models

AI summary

Overview

Research area: AI alignment / value-based alignment for Large Language Models, with connections to NLP evaluation, moral psychology, and personalization.

Technical level: Intermediate. Readers should be comfortable with embedding spaces, contrastive learning, preference/alignment concepts, and basic statistics, but the paper's framing and results are accessible without deep expertise.

Scope: A unified framework (extraction, evaluation, and steering) for aligning LLMs with multiple human value theories at controllable intensity levels, validated across ten models and four value systems.

What This Paper Is About

Existing alignment methods based on human preferences tend to capture surface-level choices rather than the deeper motivational values that explain consistent behavior. Value-based approaches are more principled, but prior work has three weaknesses: value extraction ignores the hierarchical structure of values, evaluation detects whether a value is present rather than how strongly, and no one has systematically tested whether LLMs can be steered to express values at specified intensities. VALUEFLOW closes these gaps with a single end-to-end pipeline that extracts values from text, calibrates their intensity via rankings, and steers model outputs at graded strengths.

Key Contributions

1. Hierarchical Value Embedding Space (HiVES): A two-stage trained embedding model that captures both within-theory hierarchies and cross-theory structure, unifying four heterogeneous value frameworks (Schwartz's Basic Values, Moral Foundations Theory, Rights, and Duties) into a shared representation.

2. Ranking-Based Value Intensity Evaluation + VIDB: A Plackett-Luce-based ranking framework that produces calibrated intensity scores on a [-10, 10] scale, along with a large-scale Value Intensity DataBase of value-labeled texts with intensity estimates. Ranking significantly reduces cross-model instability, sign flips, and prompt sensitivity compared to rating-based scoring.

3. Steerability with Intensity: A formal definition and empirical protocol for steering LLMs using (value, intensity) pairs, extending prior work on directional alignment to graded control. The paper identifies three steerability regimes across models and three types of per-value steerability behavior.

4. Pluralistic Alignment Analysis: Empirical characterization of how multiple values compose when steered simultaneously (additive for similar values, trade-off for opposed values, strong-anchor dominance in complex settings), plus a demonstration that value profiles improve demographic behavior prediction on OpinionQA by over 10 percentage points on some attributes.

Main Findings

• Ranking beats rating for value evaluation: Rating-based intensity scores show a mean variance of 12.6, 48% sign-flip rate, and 3.6 average prompt sensitivity, while ranking-based scoring cuts these to 2.1, 29%, and 2.3 respectively, and improves agreement with human-annotated ValueNet (86.8% vs. 82.5% sign accuracy).

• Three model steerability regimes: Weakly steerable (Phi-4, Claude-4, especially resistant to negative steering on prosocial values), moderately steerable (Qwen3, GPT-4.1, gpt-oss, Mistral-3.1), and strongly steerable (Grok-4, Gemma-3, Gemini-2.5, GLM-4).

• Three per-value patterns: Hard-to-steer values such as Conformity barely move in either direction; polarity-asymmetric values such as Hedonism respond to positive steering but resist negative steering; bi-directional values move substantially both ways, with ceiling effects suppressing positive shifts when default endorsement is already high.

• Multi-value composition laws: Similar value pairs compose roughly additively, opposed pairs produce asymmetric trade-offs, and in five-value settings the +2 target dominates while negatives mainly attenuate rather than reverse. Hierarchical composition (steering a parent value through its children) broadly matches direct steering.

• HiVES outperforms baselines: Over 20% improvement in hierarchical ranking accuracy and over 50% improvement in similarity correlation versus Qwen3-embedding and UniVar, with more disentangled value directions.

• Profile-based steering improves behavior prediction: On OpinionQA, profile-based prompts raise accuracy over default prompts across most demographic attributes — for example, Phi-4 on Religion jumps from 44.5% to 57.4%, and GLM-4 improves average accuracy from 56.8% to 59.1%. Modular Pluralism baseline performs substantially worse (under 50% in many cases).

• User-text prompts normalize extremes: Compared to explicit intensity anchor prompts like "strongly values X," prompts using real example texts from the VIDB preserve the ordering of steerability but soften large shifts and modestly nudge low-responsive values.

Methodology in Plain English

The pipeline has three stages that share a common backbone:

1. Extraction via HiVES: The team mapped texts from existing value corpora onto four value theory hierarchies using a panel of seven LLMs that votes on the correct category at each level, with human adjudication for disagreements. They then aligned all four theories into one embedding space using a two-stage training process — first hierarchical contrastive learning within each theory, then cross-theory alignment via shared "anchor" concepts (274 anchors) plus curated user-friendly value formulations.

2. Evaluation via ranking: Instead of asking an LLM to assign an absolute intensity score to a text (which is unstable across models), they ask it to rank texts relative to each other in small windows. Aggregating many rankings with a Plackett-Luce statistical model produces stable latent intensity scores. The resulting VIDB stores 10K texts per value with calibrated scores.

3. Steering with intensity control: Given a target (value, intensity) pair drawn from {-2, -1, +1, +2}, they steer the model either with explicit language cues ("strongly values X") or with example texts from the VIDB corresponding to each intensity bin, then measure the steering gain as the difference between steered and default output intensities.

The evaluation uses 500 prompts from five benchmark datasets and tests ten models across 32 values from four theories. A separate human study with 25 evaluators validates the ranking-based approach against rating-based alternatives.

Why This Matters

This paper provides the first end-to-end research infrastructure for measuring and controlling how strongly LLMs express specific human values, moving alignment beyond binary "does the model value X" questions to "at what intensity does the model value X, and can we reliably dial it up or down." It also shows that values are not a single dial — different values have different steerability profiles, and multiple values interact in structure-dependent ways (additive, trade-off, or dominance). This matters because deploying LLMs across diverse cultural and personal contexts requires more than a single fixed ethical stance.

Real-world applications:

• Personalized and culturally adaptive assistants: Build user or group profiles with HiVES and steer responses to match their value priorities rather than applying one global default.

• Policy-steerable deployment: Organizations can specify compliance-aligned value intensities (for example, "moderate privacy emphasis, strong transparency") and audit whether models honor them.

• Safety auditing: The refusal-behavior analysis and steerability regimes help identify which models can be pushed toward harmful value expressions, informing red-teaming and release decisions.

• Cross-cultural research infrastructure: Extensible to other languages and value systems (Buddhism, non-Western frameworks) as the paper demonstrates, supporting comparative studies.

Industry relevance: Any organization deploying LLMs in customer-facing or high-stakes roles needs to specify and verify value behavior. VALUEFLOW offers modular components — HiVES, VIDB, or the steering protocol can each be used independently — making it adoptable without rearchitecting entire alignment pipelines.

Future Directions

• Richer value representations: The current model treats value intensity as a one-dimensional scalar conditioned on a single target value. Extending to context-sensitive, multi-dimensional representations that capture value trade-offs within a single response is an open problem.

• Efficiency of evaluation: The ranking-based evaluator requires multiple LLM calls per output, which limits real-time or large-scale deployment. Reducing this cost without sacrificing stability is a clear next step.

• Closing the gap for weakly steerable models and values: Understanding why some models resist negative steering on prosocial values — and whether this reflects safety training, inductive bias, or architectural limits — could inform both alignment training and prompt design.

• Cross-cultural generalization: The paper shows extension to Chinese, Korean, Arabic, and Buddhist frameworks, but full validation across cultures, especially with non-Western value theories as primary rather than auxiliary coordinates, remains open.

• Safety-steerability interaction: The paper finds links between steerability and refusal behavior but does not fully characterize how safety-aligned models trade off steerability against harmfulness constraints.

Target Audience

This paper is most valuable for alignment researchers, ML engineers building personalization or policy-steerable systems, and social scientists studying how LLM behavior relates to human value frameworks. It is also relevant to AI ethics and safety teams who need empirical tools for auditing value expression across models, and to researchers in computational social science who work with Schwartz's value theory, Moral Foundations Theory, or deontic frameworks. Readers familiar with embedding models and preference alignment will get the most from the technical sections, but the conceptual framework and steerability results are readable by anyone with an interest in how LLMs represent and express human values.

Authors’ abstract

Aligning Large Language Models (LLMs) with the diverse spectrum of human values remains a central challenge: preference-based methods often fail to capture deeper motivational principles. Value-based approaches offer a more principled path, yet three gaps persist: extraction often ignores hierarchical structure, evaluation detects presence but not calibrated intensity, and the steerability of LLMs at controlled intensities remains insufficiently understood. To address these limitations, we introduce VALUEFLOW, the first unified framework that spans extraction, evaluation, and steering with calibrated intensity control. The framework integrates three components: (i) HIVES, a hierarchical value embedding space that captures intra- and cross-theory value structure; (ii) the Value Intensity DataBase (VIDB), a large-scale resource of value-labeled texts with intensity estimates derived from ranking-based aggregation; and (iii) an anchor-based evaluator that produces consistent intensity scores for model outputs by ranking them against VIDB panels. Using VALUEFLOW, we conduct a comprehensive large-scale study across ten models and four value theories, identifying asymmetries in steerability and composition laws for multi-value control. This paper establishes a scalable infrastructure for evaluating and controlling value intensity, advancing pluralistic alignment of LLMs.