Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models

AI summary

Overview

Research area: Natural language processing, specifically ethical alignment, multilingual evaluation, and parameter-efficient fine-tuning (LoRA SFT/DPO) plus weight-space task-vector arithmetic in large language models.

Technical level: Intermediate. The dilemma-benchmark and prompting results are accessible to a general reader, but the task-vector orthogonalization procedure assumes familiarity with fine-tuning, LoRA adapters, and weight-space arithmetic.

Scope: The paper builds a 12,000-instance multilingual benchmark of two-option ethical dilemmas over three value pairs, shows that compact Llama-3.2 (1B/3B) models can learn stable value preferences through LoRA fine-tuning, and demonstrates that the learned preference can be isolated as a vector in weight space and reversed on demand without retraining.

What This Paper Is About

Language models increasingly make morally loaded decisions, yet they carry hidden value preferences and follow instructions unevenly across languages. This paper asks whether a compact model (1B–3B parameters) can learn an implicit preference between two conflicting but equally virtuous principles from ethical dilemma data, and whether that preference lives in an isolatable part of the model's weights that can be flipped without retraining. The authors operationalize the question using three principles — Honesty, Justice, and Autonomy — pitted against each other in pairwise dilemmas across five languages.

Key Contributions

1. A formalized, multilingual dilemma benchmark. The authors define what an ethical stance and an ethical dilemma are, then build a 12,000-sample dataset of two-option dilemmas over three value pairs (Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty), translated into Hindi, Arabic, Spanish, and Chinese, alongside a separate human-written gold set.

2. Evidence that compact models can learn stable ethical policies. LoRA SFT and DPO training on Llama-3.2 1B/3B removes strong positional bias and reaches more than 98% accuracy on the synthetic test sets and above 90% on the human-written gold set, whereas instruction-only prompting on GPT-5-mini remains uneven across languages and value pairs.

3. A task-vector method that isolates and reverses preference. The authors extract a preference direction from a policy-aligned checkpoint, estimate and subtract an instruction-only component using a pair of models fine-tuned on opposing preferences, and recombine the residual vector to produce a model with the opposite stance at inference time.

4. A negative result on transitivity. Preference vectors for different stances are nearly orthogonal in weight space (cosine angles above 80 degrees), so task arithmetic succeeds at reversing a binary stance but fails at chaining value relations transitively.

Main Findings

• GPT-5-mini has measurable default value biases. With no policy given, it selects the Justice-aligned option in Task BC roughly 70% of the time across all five languages, and shows a similar preference for Honesty over Autonomy, with a weaker Honesty-over-Justice preference. Table 1 reports, for the first-listed guiding principle: Task AB 56.8% (English), 64.5% (Hindi), 58.8% (Spanish), 60.5% (Arabic), 62.0% (Chinese); Task BC 69.5%, 73.5%, 68.5%, 67.5%, 71.5%; Task CA 30.0%, 27.2%, 29.5%, 28.0%, 29.2%.

• Prompt steering works but is brittle. When told which stance to follow, GPT-5-mini can reverse its default bias, but performance degrades for Hindi, the lowest-resource language tested. In Task AB, accuracy for selecting Honesty over Justice falls from 64.5% to 61.8% even while being prompted with that exact policy.

• Untrained Llama checkpoints are near-random with strong position bias. Both Llama-3.2 1B and 3B behave nearly at random on the forced-choice task in zero-shot mode, mostly picking the first-listed option (and in some cases the second, primarily with Arabic). The authors attribute this to weak ethical reasoning in these raw checkpoints.

• Fine-tuning is highly effective. Five-epoch LoRA SFT eliminates positional bias and exceeds 98% accuracy across all value pairs and languages on the held-out synthetic test set; DPO shows a comparable trend with only a small drop relative to SFT.

• Learned policies generalize to human-written dilemmas. On 60 human-authored dilemmas, the SFT checkpoints reach above 90% accuracy and the DPO checkpoints show a comparable trend, indicating the models are not relying only on synthetic wording or surface cues.

• Task-vector steering preserves most fine-tuned performance. In the 1B model, transfer retains at least 93% of fine-tuned performance on Task BC and at least 80% on the other two pairs. For the 3B model, retention is at least 98% on two tasks and 85% on the third, which occurs only in Hindi. In the Observations section the authors report retention of at least 97% of full fine-tune accuracy in the 3B model on all tasks and languages except Task AB in Hindi, and the Conclusion states retention of at least 96% in most cases for the 3B model.

• Naive vector reversal fails. Simply negating the full task vector and adding it to the base model produced invalid and garbage responses, which motivated separating the instruction-following component from the preference component.

• Optimal mixing weights are moderate and bounded. Grid search over the mixing weights found the best transfer performance at γ₁ between 0.3 and 0.8 and γ₂ between 0.3 and 0.7, suggesting preference information is largely orthogonal to instruction-following ability.

• Transitivity does not hold. Attempts to chain learned pairwise value directions into a third composite stance produced near-random results after extensive grid search, because the preference vectors are close to orthogonal in weight space.

Methodology in Plain English

Building the data. The authors picked three widely used ethical principles — Trustworthiness and Honesty (A), Basic Justice or Fairness (B), and Respect for Autonomy (C) — and paired them into three conflicts: AB, BC, and CA. They hand-wrote a list of 20 social scenarios (family, healthcare, workplace, social media, and so on) and used the GPT-4o API to generate dilemmas within each scenario where both options are defensible and only a stated value ordering makes one "correct." For each stance they generated 200 dilemmas per scenario across 10 API calls of 20 dilemmas each, yielding 4,000 samples per value pair, split into 3,200 train, 400 dev, and 400 test samples. Because the generators made errors, they used Gemini-3-Flash-Preview as an automated validator that checked whether each option matched its intended value label and whether the labels matched the target stance; it proposed edits for 4,055 of the 12,000 samples, and only the post-validation dataset was used.

Translation. Each English dilemma was machine-translated into Hindi, Arabic, Spanish, and Chinese with Google Translate, with a small sample checked by back-translation and native-speaker inspection. Splits are parallel across languages. The authors are explicit that these are controlled linguistic variants, not culturally localized dilemmas, so the multilingual results test language robustness rather than cross-cultural moral equivalence.

Human gold set. Six peers each wrote 20 dilemmas for the three value pairs. Each data point was annotated by two annotators for appropriateness, correct labeling, and confidence, with 10 samples per task mixed in from the synthetic test set. Only samples where both annotators agreed were kept; disagreements were discussed and revised.

Models and training. GPT-5-mini was queried zero-shot through its public API with a minimal prompt asking for only "A" or "B," both with and without an explicit stance instruction. Llama-3.2 1B and 3B were used for the training experiments. LoRA adapters were attached only to the W_q, W_k, and W_v projection matrices in every transformer layer, with rank 4 and scaling factor 8 for the 1B model and rank 16 and scaling factor 16 for the 3B model. Training used a learning rate of 2e-4, weight decay 0.01, batch size 8, 5 epochs, and a maximum sequence length of 512. DPO used identical adapter settings, the same sequence length, batch size, and epoch count, with a regularization parameter beta of 0.1, run over three seeds (0, 1, 2) with mean and standard deviation reported. At inference, the stance is never stated; the model simply returns "A" or "B," and option order is randomized to prevent reward hacking.

Task-vector transfer. The idea is to take the weight difference between a fine-tuned checkpoint and the base model and treat it as a direction. The authors first tried simply negating that direction and adding it back to the base model, which broke generation. They concluded the difference contains both an instruction-following component and a preference component. Their remedy requires a pair of models fine-tuned on opposing preferences, whose vectors are averaged to approximate the pure instruction direction. The original preference vector is then projected onto that instruction direction and the projection is subtracted, leaving a "preference-only" vector. The final model adds the instruction vector scaled by γ₁ and subtracts the preference-only vector scaled by γ₂. The two mixing weights were tuned by grid search on the dev set, first coarsely with step 0.3 and then finely with step 0.1, over γ₁ in [0, 3] and γ₂ in [0, 2].

Why This Matters

Impact on research. The paper reframes value conflict as a controlled supervision signal rather than an underspecified prompt artifact, and provides a rare negative result — that successful preference reversal does not imply transitive composition of value directions. That bounds how far "linear geometry of values" claims can be pushed and gives follow-up work a concrete failure mode to explain. It also extends the ethical-alignment literature beyond English and beyond frontier models, and tests generalization against human-authored dilemmas rather than generated ones.

Real-world applications.

• Content moderation systems that must apply a different value ordering in different markets (for example prioritizing autonomy in one jurisdiction and fairness in another) without maintaining separate trained models.
• Policy drafting and advisory assistants that need to switch between documented ethical frameworks on demand, with the switch auditable as a single vector.
• On-device deployment of 1B–3B models, where the compute savings of PEFT and the storage savings of a shared base plus swappable preference vectors matter most.
• Multilingual products that must apply consistent moral priorities across Arabic, Chinese, Hindi, Spanish, and English, where the paper shows prompt-only steering is least reliable.

Industry relevance. The task-vector recipe gives a concrete mechanism for value pluralism: one base model plus a small set of preference directions, adjusted at inference time, instead of one fully trained model per stance. The paper is also candid that the method does not eliminate all training — it still depends on already fine-tuned preference checkpoints and on dev-set tuning of the mixing coefficients.

Future Directions

• Test whether the findings scale. All experiments are limited to Llama-3.2 1B and 3B; whether the same behavior holds in larger models is explicitly left open.
• Build culturally localized rather than merely translated data. The authors note their multilingual sets are machine-translated controlled variants, so cross-cultural equivalence of moral interpretation remains untested.
• Understand and possibly overcome the transitivity failure. Since preference vectors are nearly orthogonal (angles above 80 degrees), it is unclear whether a different composition rule, a nonlinear method, or a different projection basis could chain value relations successfully.
• Strengthen human validation. The gold set is small (60 dilemmas), so its results are preliminary evidence of transfer; the authors call for broader human and cultural validation, and for scaling beyond the three selected principles.

Target Audience

This paper is most useful to alignment and safety researchers working on value pluralism and moral decision-making in LLMs; to practitioners applying LoRA SFT or DPO to small open models; to engineers interested in weight-space editing, task arithmetic, and model merging; and to multilingual NLP researchers studying cross-lingual robustness of ethical behavior. Readers without a background in fine-tuning can still follow the benchmark construction, the baseline bias results, and the general findings, but will need some grounding in adapter-based training to follow the task-vector method.

Authors’ abstract

Large Language Models (LLMs) are increasingly deployed in applications that must weigh clashing moral values, yet even strong models exhibit hidden biases and brittle instruction-following across languages. We introduce a 12,000-instance dataset of two-option dilemmas covering pairwise three value conflicts: Honesty vs. Justice, Justice vs. Autonomy, and Autonomy vs. Honesty, along with their translations into Hindi, Arabic, Spanish, and Chinese, to probe cross-lingual behavior. Benchmarking on GPT-5-mini reveals that it consistently favors Honesty over Autonomy across all five languages when no policy is given. The Llama-3.2-1/3B models exhibit strong first-option bias; however, both plain fine-tuning and Direct Preference Optimization fine-tuning effectively remove this bias, increasing accuracy to greater than 98%. In order to decouple the effect of learning correlations in the dataset from abstract values, we propose a task vector transfer based experiment where after computing the task vectors for a direction of value preference we orthogonalize it with respect to the general instruction following vector. Our experiment shows that this method is effective in isolating the direction of the specific value preference that can successfully be used to conduct task arithmetic to obtain a model with the opposite stance.