Auditing Alignment Controllability in LLMs via Political Axes

AI summary

Overview

Research area: AI safety and ethics — specifically, the controllability (steerability) of large language models' political behavior under system-prompt framing.

Technical level: Intermediate. The core idea is intuitive (how far can you push a model's politics with a prompt?), but the paper uses variance decomposition, Spearman correlation, bootstrap confidence intervals, and displacement-versus-proximity geometry that assume some statistical familiarity.

One-sentence scope: A 63,700-response black-box benchmark measuring how far seven frontier LLMs move along political axes when subjected to twelve system-prompt ideological personas at three intensity levels, arguing that political audits should measure controllability profiles rather than static compass coordinates.

What This Paper Is About

Most political audits of LLMs place each model at a single point on a political compass and report that coordinate. But in real deployments, what matters is not where a model rests by default — it is how far, and in which directions, its answers can be steered by whoever controls the system prompt. The authors stress-test this controllability across twelve ideological personas, a baseline, and seven leading commercial models to characterize how models actually move under political pressure rather than where they sit when left alone.

Key Contributions

1. Conceptual reframing. Political coordinates are single points on a surface that instructions can move; they are not complete descriptions of deployed behavior. The paper shifts evaluation from position to a controllability profile — dispersion, directional symmetry, saturation, and refusal floors.

2. Empirical dominance of context over model identity. Within the benchmark, system-prompt framing accounts for roughly 88% of economic-axis variance and 93% of society-axis variance, while differences between models account for under 3% on both axes.

3. Diagnosis of the "asymmetry paradox." Prior audits disagreeing about whether models are more steerable left or right are reconciled once baselines are recognized as non-centered. Displacement (how far a model moved) and proximity (how close it landed to a target) rank directions differently, because the effect is geometric rather than differential compliance.

4. Cross-model convergence under extreme framing. Seven independently trained models produce correlated per-question shift vectors under authoritarian framing (mean Spearman r = 0.79, permutation p < 0.001), and convergence rises monotonically with prompt intensity.

The authors release prompts, benchmark data, and code.

Main Findings

• Context overwhelms model identity. Economic axis: η_ctx = 0.8816 vs. η_model = 0.0250. Society axis: η_ctx = 0.9313 vs. η_model = 0.0072. The government axis is the most context-driven (η_ctx = 0.9558). Model main effects stay under 3% on every axis, and low model variance is not a design artifact — rigidly aligned or refusing models would have produced flat dispersion and a large η_model.

• Interaction matters more than baseline. Adding a context × model interaction absorbs 8.4% of economic-axis variance and 5.1% of society-axis variance — three to seven times the model main effect. Model identity matters through how each model responds to each framing, not through its default position.

• Two dispersion tiers. High-dispersion group (Kimi K2: 33.32, Qwen3.6 Max Preview: 33.28, GPT-5: 31.41) moves roughly 20% more from baseline than the low-dispersion group (Claude Sonnet 4.5: 29.17, Grok-4.3: 28.75, Gemini 2.5 Flash Lite: 26.49, DeepSeek-Chat v3.1: 24.04). Tier composition is largely stable across all four axes, though magnitude varies (per-model econ and society dispersion correlate at r = 0.54, p = 0.21).

• The asymmetry paradox resolved geometrically. Mean displacement is larger toward right-economic (51.97 vs. 37.76) and authoritarian (49.30 vs. 25.50) framings. But proximity reverses the economic-axis ranking: LO_3 lands closer to its target than RO_3 (7.85 vs. 11.48). All seven models start in the Progress half of the social axis and six of seven in the Left-Progress quadrant, so rightward and authoritarian prompts simply have farther to travel. Displacement measures effort; proximity measures attainment.

• Signed-axis confirmation. Under authoritarian framing (0A_3), mean government-axis shift is −49.9 points; under libertarian framing (0L_3), +29.3 points. The asymmetry is asymmetry-of-magnitude, not asymmetry-of-compliance — no model shows a flat-response floor in the authoritarian direction.

• Saturation at ideological extremes. Under the most extreme left-economic framing (LO_3), 5 of 7 models score lower on the economic axis than under the moderate LO_2 version. Gemini 2.5 Flash Lite, GPT-5, and Grok-4.3 show reduced 2D displacement at LO_3 vs. LO_2. This non-monotonic pattern is inconsistent with pure sycophancy (which predicts uniform compliance with intensity) and incompatible with soft-refusal hedging: neutral answers drop from 37.93% at baseline to 1.92% under LO_3.

• Cross-model convergence scales with intensity. Mean pairwise Spearman correlation on per-question shift vectors rises from 0.34–0.46 at moderate intensity to 0.66–0.81 at maximal intensity, strictly increasing at every step across all four ideological families. Under authoritarian framing, about three-quarters of the 70 items receive unanimous-sign agreement across all seven models.

• Refusal floors are low but model-dependent. Refusals hold at 1.19% even when the prompt permits them. Baseline neutral-response rates span a 13× range, from 81.6% (Qwen3.6 Max Preview) to 6.4% (Gemini 2.5 Flash Lite), largely uncorrelated with compass position — a structural counter-axis that displacement-only reports hide.

• Diagonal bundling. Instruction-induced movement is not axis-independent. Models densify Left-Progressive and Traditional-Right-adjacent regions while leaving Left-Traditional and Right-Progressive quadrants sparse, even under explicitly off-diagonal prompts.

Methodology in Plain English

The researchers took a 70-item public political-orientation questionnaire (in the Political Compass/8values tradition) scoring four axes: economic, diplomatic, government, and society. Each item uses a five-label Likert scale.

For each of seven commercial frontier models, they ran the questionnaire under 13 conditions: one unsteered baseline plus 12 ideological personas. Each persona is injected as a system prompt — a 70-to-250-word paragraph written in second person ("You believe…") describing an ideological character. The twelve personas vary along two dimensions: ideological direction (economic left, economic right, social libertarian, social authoritarian) and intensity (moderate, strong, radical). A length-matched neutral control confirms prompt length is not the driver.

They ran ten replicates per model-condition cell at temperature 0.7 (a deliberately stochastic setting, making the test of robustness harder), giving 7 × 13 × 70 × 10 = 63,700 responses. Refusals, unparseable outputs, and API failures were logged and excluded rather than counted as neutral.

Analysis proceeds through three metric families: dispersion (average Euclidean distance from baseline across steered contexts), displacement vs. proximity (how far the model moved vs. how close it landed to a target extreme), and per-question shift vectors correlated pairwise across models to test coordination. Variance decomposition isolates the contributions of context, model identity, and their interaction. Bootstrap confidence intervals and permutation tests accompany the inferential claims; borderline diagnostics are flagged as exploratory.

Why This Matters

Impact on research. Static political audits may be measuring the least consequential quantity. If framing accounts for 88–93% of variance and model identity under 3%, then a compass coordinate captures a tiny slice of deployment-relevant behavior. The paper also identifies a concrete audit artifact — metric non-equivalence under non-centered baselines — that likely explains conflicting claims in prior steering literature. The proposed reporting standard (dispersion, symmetry, saturation, refusal floors) gives the field a checklist.

Real-world applications.

• Delegated agents. A user who asks an assistant to argue, draft, or triage on their behalf is asking for a model positioned at their values. Dispersion profiles indicate how reliably each endpoint can be placed there and how much further it can be pushed.
• Educational personalization. Personalized content already encodes demographic bias. A steered tutor shapes normative defaults precisely where learners are least equipped to contest them, making the allocation of default-setting authority — guardians, schools, providers, regulators — an explicit design question.
• Platform governance and induced profiles. System prompts are increasingly induced from user history and refined by automatic prompt optimization rather than written by hand. Auditing controllability at the system layer is what makes an induced default legible, and establishes what a delegation can commit its principal to.
• Procurement and model selection. Organizations can compare endpoints on reachability, saturation ceilings, and refusal floors rather than default political coordinates that may be relocated by whoever sets the instruction layer.

Industry relevance. OpenAI and Anthropic already publish political-behavior evaluations measuring refusals, asymmetric coverage, and even-handedness. This paper supplies a behavioral methodology — and a caution — for those frameworks: reporting position alone understates the variance that actually matters in interactive deployment.

A key distinction. A baseline offset can be relocated by whoever sets the instruction layer; a reachability gap, saturation ceiling, or refusal floor cannot be removed by prompting. A bias in what the profile can reach is a more serious and less tractable failure than a bias in where the profile starts.

Future Directions

1. Mechanistic grounding. The cross-model convergence in per-question shift patterns is output-level regularity, not evidence of shared internal representations. The black-box design cannot distinguish shared pretraining priors, instruction-following competence, RLHF conventions, or distillation effects. Interpretability work (linear ideology directions, "ideological depth") makes predictions the behavioral signature is consistent with — but does not confirm.

2. Saturation as a formal test. The non-monotonic endpoint pattern is reported descriptively because formal saturation tests are sensitive to unit of analysis. Developing a robust, pre-registered saturation metric is an open problem, along with distinguishing guardrail activation from diminishing marginal returns inside a model's own trained region.

3. Beyond the compass. Compass-only audits cannot detect non-compass-axis side effects — correlations between requested and unrequested behavioral changes. Extending the audit to pluralistic value profiles beyond political axes would test whether the controllability profile generalizes.

4. Open-source replication and dialogue-emergent steering. All seven models are commercial endpoints observed over a two-day window in May 2026; the dispersion and coordination patterns are hypotheses for open-source replication, not general claims about LLMs. The boundary between system-prompt steering and dialogue-emergent controllability was explicitly deferred.

Target Audience

AI safety and alignment researchers studying steerability, sycophancy, and pluralistic alignment; policy and governance analysts designing political-bias evaluations for LLMs; platform engineers responsible for personalization layers, system-prompt induction, and delegated-agent design; and education-technology developers making decisions about default framing in learner-facing systems. Statistically literate readers who want a working vocabulary for controllability profiles will benefit most; readers seeking a definitive ideological diagnosis of any specific model will not find one here.

Authors’ abstract

Political audits of large language models (LLMs) usually reduce each to one point on a political compass. But that resting point barely matters in deployment: a model must land somewhere, and what counts is how far, and in which directions, its answers can be steered. That steering runs through the system prompt: the personalization layer a platform sets, or one induced from a user's history, not necessarily written by hand. We run a dispersion-first stress test of prompt-based controllability across 12 ideological personas plus an unsteered baseline, 70 Political Compass items, ten replicates, and seven leading LLMs: GPT-5, Claude, Grok, Gemini, DeepSeek, Kimi, and Qwen (63,700 responses). Contextual framing explains roughly 88%-93% of variance on the economic and society axes, model identity under 3%: responses are highly instruction-adjustable. Models do not shift alike: some move more, and some saturate under extreme framings. Conflicting directional-steering results in prior audits resolve once baselines are recognized as non-centered: displacement and proximity diverge, so the effect is geometric, not differential compliance. Under authoritarian prompts, models produce similar shifts on the same questions. Political-coordinate audits therefore need steerability audits reporting dispersion, symmetry, saturation, and refusal floors. We release prompts, benchmark data, and code.