Towards Multi-Value Alignment via Value Localization in LLMs

AI summary

Overview

Research area: AI alignment and mechanistic interpretability of large language models, specifically localizing and controlling human-value-related behavior inside transformer attention heads.

Technical level: Advanced. The paper assumes familiarity with multi-head attention, linear probing, activation steering, and preference-optimization baselines such as DPO and MODPO.

Scope: The paper identifies sparse, value-sensitive attention heads across six value dimensions and six LLMs from three model families, then uses inference-time activation steering on those heads to control single and multiple values without parameter updates.

What This Paper Is About

Existing multi-value alignment methods rely on costly fine-tuning or preference optimization and offer little insight into how values are represented inside a model or why value trade-offs occur. This paper asks whether distinct human values can be localized to specific attention heads, and whether intervening on those heads at inference time can produce controllable alignment with one value or several values at once. The authors focus on six dimensions: harmlessness, honesty, helpfulness, complexity, verbosity, and coherence.

Key Contributions

1. A probing-based value localization framework. The authors train a separate linear probe on the last-token activations of every attention head in every layer, for each of six value dimensions, and rank heads by test accuracy to identify "value heads."

2. An empirical characterization of value heads across model families. They analyze Llama3.1-8B-instruct, Llama3.2-3B-instruct, Qwen3-8B, Qwen3-4B, Yi1.5-9B, and Yi1.5-6B, reporting sparsity, cross-model universality, middle-layer concentration for harmlessness, helpfulness, and honesty, and structured interactions and conflicts among values.

3. Causal evidence through targeted head ablation. Masking the top 15% of value heads produces large, value-specific performance drops, whereas masking an equal number of random heads produces only marginal drops, and masking value heads leaves performance on value-neutral corpora unchanged.

4. An inference-time multi-value steering method with theory. They estimate value directions as the difference between mean positive and negative activations, scale them by the activation standard deviation, and combine directions linearly across values. They provide Assumptions 1–3, Lemmas 1–2, Theorem 1, and Corollary 1 on Pareto-frontier navigation, plus comparisons against MODPO and Multi-Objective PFT.

Main Findings

• Value heads are sparse: In Qwen3-8B, fewer than 3% of heads achieve accuracies above 0.8 for harmlessness, helpfulness, coherence, and honesty. Verbosity and complexity show more diffuse patterns, which the authors attribute to their more surface-level or global nature.

• Core value heads concentrate in middle layers: Heads associated with harmlessness, helpfulness, and honesty (the "3H" values) are predominantly located in middle layers across models, consistent with prior mechanistic-interpretability findings that middle layers encode higher-level semantic features.

• The pattern is consistent across architectures and scales: Heatmaps for five additional models show similar sparsity, models within the same family (particularly Qwen) have similar distributions, and Pearson correlations across models are relatively high for harmlessness, helpfulness, and honesty.

• Ablation causes targeted collapse: For Llama3.1-8B, masking value heads reduces Malicious Instruct accuracy from 0.48 to 0.22, JailbreakBench from 0.44 to 0.20, TruthfulQA honesty from 0.61 to 0.14, HelpSteer2 helpfulness from 0.85 to 0.34, and coherence from 0.79 to 0.22, versus random-head results of 0.28, 0.35, 0.41, 0.77, and 0.69. For Qwen3-8B, honesty falls from 0.68 to 0.21. The authors report that masking identified value heads can cause performance to collapse entirely, for example Yi-1.5-6B on harmlessness.

• Random masking is comparatively harmless until it is not: Randomly masking up to 15% of heads in Llama3.1-8B-instruct has minimal impact, masking helpfulness or coherence heads causes substantial degradation, and random masking beyond 25% also causes sharp declines as functionally critical heads are included. Results are repeated across five random seeds, with statistically significant results in most cases (p < 0.05).

• Verbosity and complexity are not well localized: For these dimensions, differences between original outputs, random-head masking, and value-head masking are minimal; for Llama3.1-8B verbosity, value-head masking gives 0.96 versus 0.90 originally.

• Value heads are not generic language components: Masking them produces no observable difference from baseline or random masking on WikiText-2 and Penn Treebank.

• Values interact in structured ways: Most value pairs show weak correlations, but some pairs show relatively high positive correlation (for example harmlessness and honesty in Qwen3-4B). Masking one value's heads also degrades other values, especially among harmlessness, helpfulness, and honesty, due to head overlap. Analysis of shared-head subspaces shows most harmlessness–honesty subspaces are approximately orthogonal, with a small subset showing angles in the 90–120° range, which the authors interpret as partial representational conflict; harmlessness and helpfulness show directional similarity, while helpfulness and honesty are the most orthogonal on average.

• Single-value steering improves target values: Positive intervention on honesty heads in Yi1.5-9B raises honest responses on question answering from 62% to 69%; complexity rises from 0.71 to 0.78 for Llama3.1-8B and from 0.76 to 0.86 for Yi-1.5-9B, and helpfulness rises from 0.80 to 0.83 for Qwen3-8B. The scaling parameter α is drawn from {0.3, 0.5, 1.0}, with sensitivity analysis in the appendix and five stochastic runs reported for robustness.

• Multi-value steering works with trade-offs: On BeaverTails and HH-RLHF (200 sampled questions per dataset, λ_k = 1/K), simultaneous intervention improves most dimensions, for example Qwen3-8B honesty from 0.74 to 0.82 and verbosity from 0.86 to 0.92 on HH-RLHF, and coherence from 0.66 to 0.74. Harmlessness decreases slightly for Qwen3-8B (0.66 to 0.60 on BeaverTails, 0.70 to 0.66 on HH-RLHF), which the authors attribute to trade-offs among values. On BeaverTails with Qwen3-4B, appendix results show approximately 8%–11% gains in harmlessness, helpfulness, and honesty.

• The method outperforms training-based baselines under a one-step adaptation setting: On HH-RLHF, Llama3.2-3B reaches 0.54 harmlessness, 0.63 honesty, and 0.65 helpfulness with this method, versus 0.44/0.55/0.59 for MODPO, 0.45/0.57/0.60 for Multi-Objective PFT, and 0.47/0.57/0.59 originally. Qwen3-4B reaches 0.69/0.70/0.73 versus 0.63/0.67/0.67 for MODPO, 0.63/0.66/0.66 for Multi-Objective PFT, and 0.64/0.65/0.67 originally.

• Theory supports linear combination: Under local linearity of head activations, sparse value localization, and weak cross-value coupling, Theorem 1 shows that a convex combination of value-specific intervention directions increases each targeted value score, with cross-value terms bounded by small ε_jk, and Corollary 1 describes a continuous manifold near the Pareto frontier parameterized by the coefficients α_k.

Methodology in Plain English

The authors treat each attention head as a candidate "knob" for a value and test every knob separately.

1. Build labeled probing sets for six values. Honesty uses TruthfulQA (817 questions across 38 categories, randomly split 4:1 into train and test). Harmlessness uses Anthropic HH-RLHF filtered to single-round interactions, yielding 49,436 training and 2,612 testing samples, with chosen responses labeled harmless and rejected responses labeled harmful. Helpfulness, complexity, coherence, and verbosity use HelpSteer2, whose responses are annotated on a 5-point Likert scale from 0 to 4; the authors binarize these labels—rating 4 is positive and 0 or 1 is negative for helpfulness and coherence, while ratings 3 or 4 are positive and 0 is negative for complexity and verbosity.

2. Probe every head. For each question–answer pair, they concatenate question and answer, extract each head's activation at the last token, and train a linear probe (a sigmoid over a dot product) per head per layer. Heads with higher test accuracy are treated as more important for that value. An appendix ablation compares this QA format with a QAQ format that appends another question, and finds QA performs better.

3. Check whether the heads matter. They ablate a head by scaling its output by a small factor ε (for example 0.001), then compare masking the top 15% of value heads against masking the same number of random heads. Behavior is scored by an LLM judge, Qwen3-30B, which labels responses as 1 if they show the target value and 0 otherwise. Harmlessness is tested on JailbreakBench and Malicious Instruct; other values use their probing test sets; general language ability is checked on WikiText-2 and Penn Treebank.

4. Estimate a steering direction. For each value, the direction for a head is the mean activation on positive samples minus the mean activation on negative samples. They estimate the standard deviation of activations along that direction and shift the head activation by α times that standard deviation times the direction at inference time.

5. Combine directions for multiple values. For a set of values, each head's composite direction is a weighted sum of its per-value directions, with weights λ_k satisfying a sum-of-absolute-values bound of 1 for stability. This lets the model be steered toward different value configurations without retraining.

6. Compare against trained baselines. They compare with MODPO and Multi-Objective PFT using a one-step parameter adaptation setting, and report multiple-run results in the appendix.

Why This Matters

Impact on research: The work connects mechanistic interpretability with alignment practice. It provides evidence that human values are not uniformly distributed across a model but are concentrated in sparse, identifiable components, and it offers a testable account of why some value pairs reinforce each other while others conflict. The theoretical framing—local linearity plus sparse localization plus weak coupling—gives subsequent work a concrete set of assumptions to challenge or extend.

Real-world applications:

• Safety filtering and refusal behavior: Understanding which heads govern harmlessness could support auditing and reinforcing refusal behavior, while also exposing how easily such behavior might be suppressed.
• Customizable assistants: Different users or jurisdictions may want different trade-offs among helpfulness, honesty, verbosity, and harmlessness; inference-time coefficients allow adjustment without separate trained models.
• Model auditing and documentation: Localized value heads give auditors a concrete internal target to inspect when assessing whether a model's stated values match its behavior.
• Low-cost deployment-time adaptation: Because no parameters are updated, value behavior can be adjusted per request or per deployment without retraining.

Industry relevance: The method avoids the compute and data costs of RLHF, DPO, MODPO, or Multi-Objective PFT, and it can be applied to an already-deployed model. The reported comparison against MODPO and Multi-Objective PFT under a one-step adaptation setting, together with maintained performance on value-neutral corpora, is directly relevant to teams that need pluralistic or configurable model behavior at serving time.

Future

Authors’ abstract

Aligning large language models (LLMs) with multiple, potentially conflicting human values remains challenging, as existing multi-objective alignment methods often require costly fine-tuning and provide limited insight into the internal mechanisms underlying value-related behaviors. In this work, we investigate multi-value alignment from a mechanistic interpretability perspective by identifying value-sensitive attention heads (value heads) in LLMs. Using a probing-based value localization framework, we discover sparse sets of attention heads associated with diverse dimensions, spanning both human values (e.g., harmlessness, honesty) and stylistic preferences (e.g., coherence, complexity). Analyses across multiple LLM families reveal functional specialization and structured interactions among value heads, including conflicts between certain values. Targeted head ablations further induce substantial, value-specific behavioral changes, providing causal evidence of their functional importance. Building on these findings, we propose an inference-time activation steering strategy that enables controllable alignment with individual values or combinations of values without updating the underlying model parameters, supported by theoretical analysis. Experiments across multiple value alignment benchmarks demonstrate effective single- and multi-value control while largely preserving non-target model performance. Our findings connect mechanistic interpretability with flexible multi-value alignment, offering a foundation for more interpretable and pluralistic LLM behavior. The code is available at https://github.com/sihuo-design/ValueHeads.