People Are Not Just Their Countries. Disentangling Social Determinants of LLM Value Alignment Across Europe

AI summary

Overview

Research area: AI value alignment and AI ethics — specifically, how well Large Language Models (LLMs) reflect the stated values and opinions of different human populations, measured through large-scale survey data.

Technical level: Intermediate. The paper combines survey methodology, statistical reweighting, and predictive modelling, and it reports model names and scores without requiring deep machine-learning background to follow the main arguments.

Scope in one sentence: Using the European Social Survey, the paper measures how 10 commercial LLMs align with the stated values of respondents across 15 socio-demographic variables and country of residence, then disentangles how much of the variation in alignment is attributable to country versus individual social characteristics.

What This Paper Is About

Most prior research on LLM value alignment compares models against populations defined by national borders or broad cultural blocks, effectively treating a country as a single set of values. This paper asks whether that framing hides important divisions, by checking whether LLMs align better with some socio-demographic groups within Europe than others — and whether observed differences between countries are really just reflections of differing demographic make-up. The goal is to identify who generative models align with, not only which countries they align with.

Key Contributions

1. A first cross-national evaluation of LLM value alignment against socio-demographics, using the European Social Survey (ESS) as an alternative to the widely used World Values Survey (WVS), which the authors argue raises generalizability and data-contamination concerns.
2. Evidence that LLMs are unequally aligned across socio-demographic groups and countries, reproducing global patterns of alignment toward WEIRD (Western, Educated, Industrialized, Rich, Democratic) populations: groups that are richer, more educated, and from more Western European countries have values that the LLMs better represent.
3. A demonstration that country of residence matters in its own right, and that between-country alignment differences cannot be explained by the 15 considered socio-demographic variables alone — yet country and socio-demographics are complementary, with their combination providing by far the highest explanatory power.
4. A question-set sensitivity finding: the relative importance of country versus socio-demographics shifts depending on whether the analysis uses the broad question set or only the 21 Portrait Value Questionnaire (PVQ) items, emphasizing the role of survey design in alignment measurement.

Main Findings

• Overall alignment varies substantially by model. On the restricted question set answered by all models, overall alignment ranges from 0.581 to 0.745. On the full question set it ranges from 0.607 to 0.746 (highest: claude_opus47 at 0.746; lowest: gpt5_5 at 0.607). Bootstrap confidence intervals are all under ±0.005.

• Gender: Respondents identifying as women have higher alignment scores, with a mean difference in cross-model deviation of 0.0095 between men and women. Only for claude-opus-4-7 does this difference change sign, and only very slightly.

• Ethnicity and migration background: Differences here are not large — 0.0068 — but respondents with a Western immigration background score above average and those with a non-Western background below average. Respondents who do not identify as part of the ethnic majority show a deviation of -0.01.

• Socio-economic status: Higher financial stability across life (income decile, perceived financial difficulty, childhood financial difficulties) tracks with higher alignment. The widest gap, 0.0385, is between the highest and lowest categories of Household Income Feeling. Higher education correlates with higher alignment, with a noticeable jump from master's-level to doctoral education. Unemployed respondents tend to have lower alignment. Occupations support a "class conscious" reading: higher social class, better education, higher income, and more white-collar work align better with the models.

• Urban-rural setting: No obvious overall pattern. People living on a farm or in the countryside emerge as best aligned among domicile types, with a cross-model mean deviation of 0.0145.

• Religion: More religious people have lower alignment. The widest spread across all socio-demographics concerns religious denomination: Muslims and Eastern Orthodox respondents on one side and Protestants on the other are separated by 0.051 points. Muslims show the most negative deviation, -0.035. Roman Catholics and other Christian denominations score comparatively lower than Protestants.

• Generations: A quadratic (U-shaped) pattern appears, with models aligning worse with the youngest and oldest cohorts. This U-shape results from aggregating diverging model-specific patterns, so comparative alignment for young and old individuals depends primarily on model choice.

• Online activity: Both the groups spending the most and the least time online are best represented. The authors suggest higher alignment for heavy internet users is unsurprising since they may have authored more training text, and note the high alignment of those who spend little to no time online "could be worth exploring."

• Political interest: Models align better with more politically interested individuals, especially compared to those not at all interested.

• Countries: The spread across countries is high. Scandinavian and central European countries are comparatively best reflected; some Balkan and Baltic countries are least captured. Between Bulgaria (lowest) and Sweden (highest) the difference in mean deviation is 0.0896, larger than within any single socio-demographic factor.

• Composition does not explain country gaps: After inverse propensity weighting to equalize socio-demographic distributions across countries, the between-country standard deviation of country mean alignment remained mostly unchanged for every model, and reweighted deviations did not move much closer to zero. Between-country differences therefore cannot be explained by the socio-demographic compositional differences considered.

• Predictive modelling — combined covariates work best: Using country and socio-demographics together in a gradient boosted model explains a substantial share of individual alignment variance. claude-opus-4-7 reaches the highest test R² of 0.428; even deepseek_V4, the lowest, reaches 27.8%.

• Predictive modelling — country alone rivals all socio-demographics: On the full question set, country of residence as a stand-alone variable explains at least as much variance as the full set of 15 socio-demographic factors for all LLMs. Country alone explains between 7.3% (mistral_lg) and 27.8% (claude_opus46), with the remaining models falling between 15% and 22%.

• The PVQ sub-analysis looks different: On the 21 PVQ questions, country alone explains a much smaller proportion of variance for the mistral-medium-3-5 variants and the gpt and deepseek families, both in absolute terms and relative to socio-demographics. Only for claude-opus-4-6 is country's explanatory power much higher than the socio-demographics-only models. The authors interpret this as showing that no single definition of alignment is more correct — broader question sets may capture a country's political climate and media landscape, while excluding them may make people with similar abstract values but opposing practical opinions appear "aligned."

• Robustness: A shadow analysis on answer extremeness found that respondents' tendency to pick Likert options far from the midpoint does not explain the core findings, though it may play a role for claude_opus_4-7 and mistral-lg, which tend to answer at the midpoint.

Methodology in Plain English

The researchers took the 11th wave of the European Social Survey, conducted between 2023 and 2024 across 29 European countries and Israel, covering 50,116 respondents aged 15 and older in private households. They selected 47 value- and opinion-related questions that were not country-specific (the paper notes 53 questions are actually considered, including a subset of 9 questions corresponding to 3 conceptual questions). Of these, 21 form a shortened Portrait Value Questionnaire based on Schwartz's theory of basic human values.

They prompted 10 commercial LLMs from four providers — OpenAI and Anthropic (U.S.), Mistral (Europe), and DeepSeek (China) — with each survey question 20 times, using no system prompt and default parameters, and prompting only in English. Answers were mapped onto the ESS Likert scales with a rule-based approach whose accuracy was 97.6%, validated by hand-annotating a 127-answer subset of non-single-number responses. The majority vote across the 20 calls became each model's answer.

Alignment was scored per person, per model, per question as 1 minus the normalized distance between the human answer and the model answer on the Likert scale, giving a value between 0 and 1, then averaged across questions. Group-level scores are reported as deviations from the overall population mean, averaged across all 10 models. Uncertainty was estimated with 5,000 bootstrap samples accounting for ESS sampling uncertainty, with a secondary analysis also accounting for LLM answer variability.

To test whether country differences were really demographic composition effects, they applied inverse propensity weighting so each country's socio-demographic distribution matched the pooled distribution, with propensity scores clipped at 0.01 (robustness checks dropped scores at the 0.01 and 0.05 thresholds). Finally, they predicted individual alignment scores using ordinary least squares regression and gradient boosted tree ensembles (XGBoost) under three covariate sets — country only, socio-demographics only, and both — comparing test R² over 10-fold cross validation, weighted by ESS post-stratification weights.

Why This Matters

Impact on research: The paper argues that alignment research has conflated the opinions of diverse populations under national labels, creating blind spots about social stratifiers beyond nationality. It also introduces the ESS as an alternative benchmark to the heavily used WVS, whose decades-long history makes contamination likely and whose wave used in prior work was collected between 2017 and 2022, partly during the Covid-19 pandemic. Methodologically, it shows that the choice of question set changes conclusions about what drives alignment, which has implications for how alignment benchmarks are designed.

Real-world applications:

• Content moderation and information retrieval systems that embed LLMs opaquely, where unequal value representation has direct consequences for whose speech and views are treated as normal.
• AI-assisted advice and information seeking, where users may trust and depend on model output, and where systematically worse alignment for lower-income, less-educated, unemployed, or religiously non-majority users means those groups receive less representative guidance.
• Sensitive design and localization of AI products in European markets, where the paper shows country of residence shapes alignment independently of demographics.
• Survey and benchmark design for evaluating AI systems, since the PVQ versus broader question-set comparison shows results can flip depending on which questions are asked.

Industry relevance: The paper shows overall alignment differs markedly by vendor and model, and that country-level variance survives demographic reweighting — a signal that model developers cannot fix representational gaps purely through demographic balancing. The code is released at https://github.com/aida-ugent/LLMs-x-ESS.

Future Directions

• Testing whether richer or additional socio-demographic variables could account for the between-country differences that the 15 considered variables plus reweighting failed to explain.
• Investigating the unexpected finding that people who spend little to no time online have higher alignment scores, which the authors flag as worth exploring.
• Examining the extremeness of answers further, particularly for claude_opus_4-7 and mistral-lg, which tend to answer at the Likert midpoint, and whether survey response styles rather than values drive part of the patterns.
• Reconciling the divergent results across question sets — specifically why country dominates on the broad question set but not on the PVQ for most model families — to build a clearer theory of which kind of alignment a benchmark is actually measuring.

Target Audience

This paper is most useful for AI alignment and AI ethics researchers, especially those working on pluralistic alignment and representativeness; social scientists and survey methodologists interested in how LLMs relate to human value data; and AI policy, fairness, and product teams who need evidence about which user populations their models represent least well. Readers from the ESS or WVS research communities will also find relevant methodological discussion about measurement invariance and cross-national comparison.

Authors’ abstract

As Large Language Models (LLMs) are increasingly used as a primary source of information and advice, understanding their alignment to humans in terms of values becomes a pressing concern. A growing literature has leveraged large scale surveys to investigate to what extent LLMs' and humans' stated values and opinions align. With limited exceptions, studied populations have been defined country borders or cultural bounds. Yet, this focus neglects the role that socio-demographic divides may play for value alignment disparities. Relying on the European Social Survey, we address this knowledge gap by considering value alignment displayed with respect to 10 prominent commercial LLMs in terms of 15 socio-demographic variables as well as country of residence. Our analyses reveal that LLMs are indeed unequally aligned to the values of different socio-demographic groups, notably those defined by education, income, occupation and religion. When examining alignment at the individual level, a respondent's country, taken as a stand-alone variable, explains a substantial amount of variation that is on par with the full set of considered socio-demographics. Further disentangling the respective role of country-level and socio-demographic factors, we find they are complementary in explaining value alignment patterns, with their relative weights varying across the subset of questions considered.