VALCON: Benchmarking and Enhancing Value Consistency in Large Language Models via Dual-Role Variation

AI summary

Overview

Research area: Natural Language Processing — value alignment evaluation for large language models (LLMs), with a mechanistic interpretability and fine-tuning component.

Technical level: Advanced. The paper combines benchmark construction, behavioral evaluation, layer-wise representation probing, and a parameter-efficient intervention method. Readers should be comfortable with the generator/judge (LLM-as-a-judge) paradigm and hidden-state analysis.

Scope: The paper introduces VALCON, a dual-role data construction framework, builds the three-part Value Alignment Benchmark (VAB) suite from 158 expert-defined value principles, evaluates 28 LLMs, and adds a representation-level diagnosis and mitigation method for cross-role inconsistency (arXiv:2610.21304v1, 15 Oct 2026).

What This Paper Is About

Most evaluations of whether LLMs are value-aligned only check whether their outputs look compliant. Modern agent pipelines, however, often use the same model both to produce content and to judge whether that content is acceptable. This paper asks whether a model preserves its own intended value label when it switches from the generator role to a blind judge role with the generation context cleared — a property the authors call value consistency — and shows that many models do not. It then diagnoses the representational roots of this failure and proposes a fine-tuning intervention to reduce it.

Key Contributions

1. A new alignment dimension and benchmark suite: The paper formalizes value consistency as distinct from normative correctness and operationalizes it through the VALCON framework, yielding the Value Alignment Benchmark (VAB) suite with three tracks — VAB-Consistency (158 principles), VAB-Boundary (15,800 samples), and VAB-Challenge (710 samples).
2. Large-scale behavioral evidence: Evaluation of 28 LLMs (24 in the main generation–verification loop, 4 additional models used only for VAB-Challenge), from 0.6B to 671B parameters, exposes what the authors call a pervasive compliance bias — models with highly compliant outputs often fail to align judgments with their own generations.
3. Mechanistic analysis: The authors define Representation Consistency Measurement (RCM) to quantify layer-wise hidden-state drift between generator and judge roles, and report a role-dependent representational decoupling associated with the behavioral asymmetry, supported by UMAP visualizations.
4. A mitigation pipeline: A representation-directed intervention, Self–Other Overlap (SOO) regularization, applied via parameter-efficient (LoRA) fine-tuning on VAB-Boundary, improves violating-case accuracy while largely preserving general capability on MMLU.

Main Findings

• Cross-role inconsistency is widespread. Qwen3-Next-80B-A3B-Instruct reaches only 71.04% Self-Review Consistency (SRC), meaning 28.96% of cases do not preserve the intended generation target under context-cleared judgment. Raw SRC ranges across the 24 evaluated models from 99.80% (GPT-5.1, Chinese prompt) down to 71.04% for that model, with Llama-3-8B-Instruct at 85.56%.
• Explicit reasoning models tend to be more consistent. Qwen3-Next-80B-A3B-Thinking reaches 98.91% SRC versus 71.04% for its Instruct counterpart. The authors present this as an observational gap suggesting deliberation may help, not as proof that chain-of-thought is the sole causal factor.
• "Deceptive consistency" from violation-generation avoidance. Qwen3-14B reaches 93.49% raw SRC but only 67.55% Balanced SRC (B-SRC) with a 9.23% V-Ratio (violating-case ratio; the ideal is 50%). GPT-5.1 and DeepSeek-R1 generate more balanced distributions (46.45% and 48.12% V-Ratio). The authors therefore recommend reporting V-Ratio and B-SRC alongside SRC.
• Self-reported confidence is largely uninformative. Across the 24-model analysis, mean confidence is nearly uncorrelated with raw SRC (Pearson r = 0.064). At τ = 0.95 the analysis retains 89.4% median coverage with a median τ-SRC of 94.4%, and the thresholded ranking stays close to the raw ranking (Spearman ρ = 0.957). Effective-SRC correlates more with SRC (r = 0.712) than with coverage (r = 0.550).
• A sharp performance cliff between clear-cut and ambiguous cases. On VAB-Boundary, top models sit near ceiling (Doubao 99.87%; Llama-3-8B 99.63%; Qwen3-4B 99.34%; Qwen3-0.6B 83.60%), but on VAB-Challenge accuracy on violating samples drops sharply. For example, Doubao falls to 86.06% overall with 72.68% on violating samples (Δ = 13.81 overall, 27.09 on violating samples), and Qwen3-0.6B falls to 48.99% overall with 6.52% on violating samples.
• High compliant accuracy masks low violating accuracy on VAB-Challenge. Nearly every model in Table 5 scores far higher on compliant samples than violating ones — Qwen3-Max leads overall at 96.34% (95.49% compliant, 97.18% violating), while Gemini-2.5-Flash scores 99.15% on compliant but 61.41% on violating samples.
• Consistency is uneven across value dimensions. Fine-grained analysis across 16 dimensions (Figure 6) shows lower-capacity models degrade systematically on abstract principles such as Liberty and Human Rights, while stronger models keep more balanced profiles.
• Disagreements have mixed sources. A blind audit of 200 stratified VAB-Consistency disagreements attributes 164 (82.0%) to generator-side target/content mismatch, 30 (15.0%) to judge-side reversal, 4 (2.0%) to boundary ambiguity, 1 (0.5%) to refusal or evasion, and 1 (0.5%) as unresolved. In the Violating→Compliant direction, 141 cases are generator-side realization failures and 6 are genuine judge-side reversals.
• The intervention helps, but unevenly. Relative to SFT-only, SOO-FT improves violating-case accuracy by 5.07, 18.31, and 3.66 points for Qwen3-0.6B, Qwen3-1.7B, and Llama-3-8B-Instruct; relative to matched weighted cross-entropy (γ = 0) the gains are 19.44, 0.28, and 3.38 points. Reduced RCM alone does not explain the gains, so the authors treat RCM as an associative signal and the benefit as model-dependent.
• Quality control supports the labels. Blind review of 100 validation samples yields 92.3% agreement with Cohen's κ = 0.86.

Methodology in Plain English

The core idea is a two-step role swap. First, a model is given a value principle and told to produce a query–response pair with a specified target label (compliant or violating); it must produce roughly balanced numbers of both. Second, the generation context is wiped, and the same model — now acting as an anonymous judge — reads the pair and predicts a label, a confidence score, and a rationale. If the judge's label matches the label the generator was asked to produce, the case counts as consistent; if not, it counts as a cross-role failure. Notably, the "gold" label here is the requested generation target, not an externally adjudicated moral truth.

To build the benchmark, the authors synthesized Maslow's hierarchy, Scheler's phenomenology, and Schwartz's basic values into a four-tier taxonomy of 158 fine-grained principles grouped under three pillars: Survival Support, Social Order, and Civility and Progress. They validated the principles by mapping model behavior onto the Inglehart–Welzel Cultural Map across Chinese and English tracks in three national contexts (China, US, Portugal). Three tracks were then derived: VAB-Consistency (each model generates 100 roughly balanced pairs per principle), VAB-Boundary (15,800 high-confidence samples where the source model's target agreed with its own cleared judgment at confidence ≥ 0.95), and VAB-Challenge (710 class-balanced edge cases retained only when GPT-5.1, Doubao-Seed, and DeepSeek-R1 reached consensus).

For diagnosis, the authors compute RCM — the mean squared distance between generator and judge hidden states at each layer, using the final non-padding token — and visualize final-layer states with UMAP. For mitigation, they compute a Self–Other Overlap loss that pulls together the hidden states of self-generated and external trajectories at the bottleneck layer where RCM peaks, and add it to LoRA fine-tuning on VAB-Boundary data, comparing against SFT-only and class-weighted cross-entropy controls. The three fine-tuned models are Qwen3-0.6B, Qwen3-1.7B, and Llama-3-8B-Instruct. Generation used temperature 0.7 and verification temperature 0.0.

Why This Matters

Impact on research. The paper reframes alignment evaluation from a single output-level property into a reliability property that spans roles, and shows the two are dissociable: Qwen3-Next-80B-A3B-Instruct scores 94.23% accuracy on VAB-Challenge yet only 71.04% SRC on its own generations. It also supplies a reusable evaluation–analysis–mitigation template and a public benchmark (code and data at the linked GitHub repository).

Real-world applications.

• Auditing LLM-as-a-judge systems used in automated evaluation pipelines, where a model grading its own or similar outputs silently reinterprets its own intent.
• Validating agentic loops in which one model both proposes actions and checks their acceptability — inconsistency there is a direct failure mode rather than a stylistic quirk.
• Safety and red-teaming reviews that currently rely on compliance-only metrics and may overestimate reliability on contested cases.
• Model selection for deployments that require stable judgments under normative ambiguity, such as content moderation or policy screening.

Industry relevance. Because several top-scoring models show near-perfect boundary behavior alongside large drops on ambiguous violating cases, product teams choosing between models on compliance benchmarks alone may be selecting on a metric that hides brittleness. The finding that self-reported confidence is nearly uncorrelated with consistency (r = 0.064) is directly actionable for anyone gating decisions on model confidence.

Future Directions

• Native multilingual and culturally grounded principles, since the current taxonomy and prompts are bounded by source models' prompt-following and the validation shows sensitivity to national context rather than cultural universality.
• Long-tail scenarios and conflicts among multiple principles, which the authors identify as unaddressed by the current single-principle setup.
• Causal rather than associative accounts of representation geometry — the UMAP entanglement evidence is explicitly framed as qualitative, and lower RCM did not reliably track higher violating-case accuracy.
• Stronger mitigations that generalize: SOO's incremental benefit over class weighting was substantial for Qwen3-0.6B and Llama-3-8B-Instruct but negligible for Qwen3-1.7B (0.28 points), leaving open how to make representation-directed training reliable across architectures, and how to calibrate self-reported confidence so τ-SRC and Effective-SRC can carry more weight.

Target Audience

Alignment and safety researchers building benchmarks or studying LLM-as-a-judge reliability; mechanistic interpretability researchers interested in role-dependent representation differences; ML engineers deploying self-evaluating agents or automated grading pipelines; and evaluation practitioners who need to know why compliance-only metrics can mislead. Readers unfamiliar with hidden-state probing will still follow the benchmark and results sections, but the intervention portion assumes some fine-tuning background.

Authors’ abstract

Current value alignment evaluations for large language models (LLMs) heavily rely on output compliance, overlooking whether LLMs maintain value consistency when alternating between generator and evaluator roles. Such consistency is increasingly important because modern LLM agents and alignment pipelines often rely on the same LLM to both generate actions and evaluate their acceptability. To mitigate this limitation, we propose VALCON, a role-driven data construction framework that operationalizes value consistency into three benchmark suites (VAB-Consistency, VAB-Boundary, VAB-Challenge) via dual-role variation. By systematically pairing an LLM's generative behavior with its evaluative judgments under identical principles, VALCON synthesizes target evaluation data that exposes hidden value discrepancies. Experiments on 28 LLMs reveal pervasive cross-role inconsistency: models with highly compliant outputs frequently fail to align their judgments with their own generations. To understand and address this gap, we perform mechanistic probing that uncovers a role-dependent representational decoupling associated with this behavioral asymmetry. Guided by these insights, we propose a representation-directed intervention that effectively mitigates consistency failures. Our work establishes value consistency as a distinct alignment dimension and provides a complete evaluation-analysis-mitigation pipeline. Code and data are available at https://github.com/tjunlp-lab/VALCON.