SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization

AI summary

Overview

Research area: Large language model alignment, specifically multi-objective alignment where trustworthiness objectives (harmlessness, honesty) conflict with helpfulness in high-stakes queries.

Technical level: Intermediate. The paper assumes familiarity with preference optimization (DPO, SimPO, RLHF), Pareto fronts, and self-alignment pipelines, though the core idea of an ordered "trustworthy-before-helpful" priority is explained conceptually.

Scope: The paper introduces priority alignment as a new alignment objective and proposes Self-Priority Alignment (SPA), a fully unsupervised framework that improves helpfulness while maintaining harmlessness and honesty across Llama-3.1-8B-Instruct and Mistral-7B-Instruct, without human-annotated data.

What This Paper Is About

In high-stakes queries such as self-harm, legal, or medical questions, an LLM must be trustworthy (harmless, honest) and still genuinely helpful, but these two goals frequently pull against each other. Existing multi-objective alignment methods try to balance the two with fixed or heuristically tuned weights, which the authors argue is context-insensitive, lacks explicit safety constraints during optimization, and suffers from scarce annotated data capturing real trade-offs. The paper's goal is to replace balancing with a strict ordering: optimize helpfulness only after harmlessness (or honesty) has met an acceptable threshold.

Key Contributions

1. A new alignment objective, priority alignment, which formulates alignment as an ordered (lexicographic) optimization over multiple objectives, avoiding explicit weight tuning and giving more interpretable control in high-stakes settings.
2. Self-Priority Alignment (SPA), a fully unsupervised framework combining self-evaluation, dual-objective filtering, and lexicographic preference learning, requiring no human-labeled data. It uses diverse sampling with self-refinement, dual-criterion denoising, lexicographically ordered preference-pair construction, and an uncertainty-weighted SimPO loss.
3. Extensive experiments across diverse high-stakes alignment settings, showing SPA improves helpfulness while maintaining strong safety guarantees and outperforming supervised (SFT) and unsupervised (Reward Soups, Self-Criticism) baselines, plus variant baselines using standard SimPO and DPO objectives.
4. Evidence that alignment gains do not cost general utility, with MTBench and MMLU evaluations on the aligned models.

Main Findings

• SPA improves alignment across all metrics. All SPA variants outperform the Vanilla and SFT-tuned models in most evaluation settings. On Mistral-7B-Instruct, the full SPA model reaches 9.76 harmlessness and 8.39 helpfulness on SafeRLHF, 7.27 harmlessness and 7.44 helpfulness on WildGuard, and 7.18 honesty and 7.82 helpfulness on HoneSet. On Llama-3.1-8B-Instruct, SPA reaches 9.90 harmlessness and 7.14 helpfulness on SafeRLHF, 8.85 harmlessness and 6.22 helpfulness on WildGuard, and 7.75 honesty and 7.83 helpfulness on HoneSet.
• Pairwise win rates favor SPA. In GPT-4o-judged pairwise comparisons, SPA shows higher win rates, including 86% on HoneSet helpfulness.
• Joint modeling of pairwise uncertainty improves the trade-off. The full SPA, which combines SimPO length normalization with uncertainty-aware weighting, consistently achieves the best trade-off; on Mistral-7B-Instruct HoneSet it takes top scores on both honesty (7.18) and helpfulness (7.82).
• SPA outperforms multi-objective baselines, with one exception. Against Self-Criticism and Reward Soups at ratios 6:4, 7:3, 8:2, and 9:1 on Llama-8B-Instruct (SafeRLHF), SPA scores 9.90 harmlessness and 7.14 helpfulness, and leads on the weighted HH metrics at lambda 5 (9.44), 10 (9.65), and 20 (9.77). It slightly underperforms Self-Criticism on the pure helpfulness metric (7.14 vs 7.68), which the authors attribute to Self-Criticism's weaker emphasis on harmlessness.
• Generalization to unseen safety-critical data. Trained only on SafeRLHF, SPA achieves 9.80 harmlessness and 6.35 helpfulness on JailbreakTrigger, and 9.29 harmlessness and 5.26 helpfulness on WildGuard, outperforming Vanilla (9.07/4.99 and 8.22/6.11) and SFT (8.91/5.23 and 8.33/6.08) on harmlessness.
• General utility is preserved. On MTBench, SPA improves over Vanilla in three out of four cases, with gains up to +2.52% (Mistral-7B-Instruct + WildGuard). On MMLU, differences are minimal, fluctuating around plus or minus 2%.
• Iterative refinement helps, with diminishing returns. On WildGuard with Llama-3.1-8B-Instruct, both second-iteration strategies beat the single-iteration baseline; reusing the same prompts is better, especially on helpfulness (6.49 vs 6.35). Further iterations beyond the second stabilize rather than improve.
• Denoising matters. Removing the denoising component (labeled "w/o NF") on SafeRLHF causes drops in both helpfulness and harmlessness exceeding 0.1 in all cases.
• Weak-model/strong-model alignment degrades with high-variance samples. Using the RV coefficient between Mistral-7B-Instruct and GPT-4o across a subset of 400 WildGuard samples, the coefficient fluctuates when fewer than 20% of samples are retained and drops sharply once more than 32% are included.
• Judge reliability validated by humans. GPT-4o agreement with human annotations reaches up to 91% and 94% for harmlessness and 89% and 92% for helpfulness on Llama-3.1-8B-Instruct and Mistral-7B-Instruct respectively.
• Results hold under a second judge. With Claude 3.5 Sonnet as judge, SPA on Llama-3.1-8B-Instruct scores 9.48 harmlessness and 7.13 helpfulness on SafeRLHF and 8.22 harmlessness and 6.40 helpfulness on WildGuard; on Mistral-7B-Instruct it scores 9.42 harmlessness and 8.37 helpfulness on SafeRLHF and 5.19 harmlessness and 5.22 helpfulness on WildGuard.

Methodology in Plain English

The paper reframes alignment as a lexicographic optimization: first satisfy the primary objective (harmlessness or honesty), then, subject to that threshold, optimize the secondary objective (helpfulness). Because LLMs are highly non-convex and high-dimensional, the classical sequential procedure cannot be applied directly. The authors approximate it by combining two ideas: enumerating Pareto-optimal trade-offs implicitly, and learning from pairwise preferences.

SPA works in four stages, all driven by the model being aligned rather than by human labels:

1. Diverse sampling with self-refinement. For each prompt, the model generates several candidate responses using high-temperature sampling and prompt variation. Each candidate is self-scored on the primary objective and the secondary objective using scoring functions derived from an AI constitution. Instead of refining responses one by one, the model produces a single improved response that incorporates all candidates and their scores, and that refined response is rescored.
2. Dual-criterion denoising. Two filters remove unreliable data. Consistency-driven denoising keeps only prompts where the refined response strictly beats every candidate on both objectives; if none qualifies, the refined response is discarded. Informativeness-driven denoising computes the covariance matrix of the sampled scores per prompt and keeps only responses whose score variance lies in an acceptable range (determinant greater than 0 and at most a threshold), discarding prompts that are too unstable or too uninformative.
3. Lexicographic preference-pair construction. Surviving responses are turned into pairs where the preferred response is better on the primary objective, or equal on the primary objective and better on the secondary one. A margin enforces a minimum total score difference between the two responses.
4. Uncertainty-guided SimPO. The model is fine-tuned with a SimPO-based loss in which each pair is weighted by the ratio of its total score gap to the batch mean gap, raised to a hyperparameter power. This emphasizes high-confidence, high-gap decisions. SimPO is chosen because it normalizes reward by response length, preventing length from being mistaken for helpfulness.

Experimental setup: Training prompts number 300 each for SafeRLHF and WildGuard and 400 for HoneSet; evaluation uses 500 test prompts per dataset. Baselines include Vanilla, SFT on preferred samples only, Reward Soups at harmlessness-to-helpfulness ratios of 6:4, 7:3, 8:2, and 9:1, Self-Criticism, and SPA variants using standard SimPO or DPO objectives. Judging uses GPT-4o and Claude 3.5 Sonnet with pairwise and score-based assessment, validated by human annotators. Training was full fine-tuning on 8 NVIDIA H100 (80G) GPUs.

Why This Matters

Impact on research. The paper argues that alignment objectives often conflict and cannot be adequately handled by weight tuning, and that fixing a priority order is a more principled alternative. It connects preference learning to Pareto dominance and lexicographic optimization, offering a template for structured multi-objective alignment. The authors state SPA was accepted by AAAI 2026 as an oral presentation.

Real-world applications (as motivated by the paper):

• Mental health and self-harm queries, where a generic refusal can feel dismissive but unsafe suggestions are unacceptable.
• Medical and legal advice, where harmlessness must be established before helpful guidance is given.
• Honesty-critical questions, where the model must acknowledge uncertainty or lacking information before being useful.
• Broader generation and reasoning settings, such as long-form generation and tool-augmented reasoning, where the paper suggests structured alignment priorities are essential for controllable behavior.

Industry relevance. SPA requires no human-annotated preference data beyond a seed set of harmlessness- or honesty-related queries, which reduces the labeling cost that makes high-stakes alignment expensive to maintain. Because the priority structure is explicit rather than encoded in tuned weights, it also offers a more interpretable control mechanism for deploying models in regulated or safety-critical products.

Future Directions

1. Extending priority alignment beyond harmlessness and helpfulness. The paper proposes its lexicographic principle as broadly applicable to other conflicting objectives, such as coherence versus creativity and efficiency versus completeness, but only demonstrates it on harmlessness/honesty versus helpfulness.
2. Understanding iteration dynamics. The authors observe diminishing returns beyond a second iteration and better results from reusing the same prompts rather than new ones; why refinement depth helps more than breadth is left as an open question worth further study.
3. Scaling and sensitivity analysis. The paper states that the analysis of sensitivity to the number of training samples is provided in the Appendix; broader study across more models, datasets, and sample budgets remains open.
4. Robustness of self-evaluation for weak models. The denoising design is motivated by observed misalignment between weak-model and strong-model judgment on high-variance samples, raising the question of how SPA behaves when the aligning model's self-scoring is substantially weaker.

Target Audience

Researchers and engineers working on LLM alignment, safety, and preference optimization who need to handle conflicting objectives without hand-tuned trade-off weights. It is also relevant to practitioners deploying models in high-stakes domains such as health, legal, and safety-critical assistance, and to readers interested in unsupervised or self-alignment pipelines that reduce reliance on human annotation. Readers should have some grounding in DPO/SimPO-style preference learning and multi-objective optimization concepts.

Authors’ abstract

In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering: optimization of helpfulness is conditioned on first meeting trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we introduce Self-Priority Alignment (SPA)-a fully unsupervised framework that generates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine-tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperforming strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications.