Large Language Models Develop Novel Social Biases Through Adaptive Exploration

AI summary

Overview

Research area: AI safety and fairness, specifically the emergence of novel social biases in large language models (LLMs) during multi-turn, stateful decision-making — drawing on paradigms from social psychology and the economics of statistical discrimination.

Technical level: Intermediate. The experimental design is conceptually simple to follow, but the paper assumes some familiarity with multi-armed bandit problems, exploration-exploitation trade-offs, chain-of-thought prompting, and information-theoretic metrics such as entropy and Jensen-Shannon divergence.

Scope: The paper replicates an iterative hiring paradigm from psychology on a broad set of LLMs to show that models invent new demographic stereotypes from noisy feedback rather than merely reproducing human biases, and it tests a range of interventions intended to reduce that stratification.

What This Paper Is About

Most work on LLM bias tries to remove stereotypes already present in training data. This paper argues that this is insufficient, because LLMs operating as agents across many turns can generate entirely new biases about groups that have no real differences between them, simply as a side effect of exploring too little and exploiting early observations too aggressively. The authors ask whether this happens in LLMs, how severe it is compared to humans, and what kinds of interventions can counteract it.

Key Contributions

1. Demonstrating emergent, not inherited, bias. The authors show that LLMs develop novel stereotypes about four fictional demographic groups (Tufa, Aima, Reku, Weki) that do not exist in pretraining data and for which no group has an actual advantage, since every candidate–job pair succeeds with probability Bernoulli(0.9).

2. Quantifying stratification with three complementary metrics. They introduce the stratification index (SI), between-group divergence (BGD), and the group assignment stochasticity index (GASI) to separate how strongly groups concentrate in job classes, how much groups' allocations diverge from each other, and how consistent those patterns are across independent runs.

3. Showing a capability–bias paradox. Newer and larger models within every tested family stratify significantly more than their predecessors, even as they improve on single-turn benchmarks such as BBQ — reasoning models like OpenAI o3 (SI = 1.83, BGD = .80) and DeepSeek-R1 (SI = 1.41, BGD = .71) were the most stratified of all.

4. Systematically testing interventions. The authors evaluate system-level, structural, and prompt-steering interventions, finding that most have limited effect while explicitly incentivizing diversity in the prompted objective is the most robust way to reduce stratification.

Main Findings

• LLMs segregate more than humans. Human participants produced SI = .84 (95% CI [.79, .89]) and BGD = .56, well above fair random assignment (SI = .25, 95% CI [.22, .29]; BGD = .29) and Thompson sampling with a uniform prior (SI = .61, 95% CI [.52, .69]; BGD = .47). All frontier LLMs averaged SI = 1.39 and BGD = .69, exceeding the human level.

• The most stratified models were specific and identifiable. Among non-reasoning models, Claude Sonnet 4 with direct prompts stratified the most (SI = 1.79, 95% CI [1.70, 1.87]), while Qwen 2.5-72B with chain-of-thought was closest to human levels (SI = .89, 95% CI [.72, 1.05]).

• Newer and larger models stratify more within families. Across {Claude, GPT, Gemini, Llama 3.2, Llama 4, Qwen 2.5}, both SI and BGD increased significantly with model recency and scale; Claude 4 Sonnet's SI was more than eight times Claude 3 Haiku's under direct prompting.

• Improved benchmark performance does not translate to less emergent bias. Models scoring higher on the BBQ bias benchmark segregated more extremely in this paradigm — the opposite of the trend on single-turn evaluations.

• The biases are learned within each run, not drawn from training data. Group–job assignments were highly stochastic across runs (mean GASI = .52, versus .47 for humans, with model-level values in Table 1 spanning 0.30 to 0.63). Ablations attribute this stochasticity to the binary success/failure outcomes of candidates rather than to prompt or sampling variation, and the models showed no prior biases toward the fictional groups.

• Chain-of-thought, temperature, and context compression are largely ineffective. CoT reduced SI in most frontier models but often not statistically significantly, and Qwen 2.5-72B fell within human range. Raising temperature to 1.5 produced no statistically significant reduction for Gemini 2.5, GPT-4o, or Llama 4 Maverick at α = 0.05 (Qwen 2.5-72B dropped significantly at p = 0.04, but to SI = 0.91, still high), and CoT outputs devolved into gibberish at T = 1.5 and T = 1.2. Replacing full dialogue history with a running count of assignments and successes gave SIs of 1.39, 1.25, and 1.09 for GPT-4o, Claude 4 Sonnet, and Llama 4 Maverick, all far above the human baseline.

• Stratification generalizes across grounded settings but not to abstract ones. In refugee resettlement and military conscription variants, average SIs were 1.13 and 1.26 respectively across five frontier models and both prompt types. Removing the cover story entirely caused models to collapse to repeated selection of one initially successful action, with demographic-wise hiring entropy (out of 2) of 0.62 for GPT-4o and 0.42 for Llama 4 Maverick in the abstract setting versus 1.90 and 1.64 in the original setting.

• Lower success rates help, but only artificially. Reducing all success probabilities from 0.9 to 0.1 reduced stratification, with larger gains under CoT (Llama 4 Maverick: SI = 1.23 with direct prompting versus 0.31 with CoT), but only GPT-4o's direct and Claude 4 Sonnet's CoT assignments fell below the random-assignment threshold of SI = 0.291 ± 0.053.

• Realistic, job-specific success probabilities erode those gains. Using success rates elicited from GPT-4o itself (ranging 6–87% across jobs), SIs rose to 0.82 (direct) and 0.60 (CoT), far worse than the ideal levels reached at p = 0.1.

• Models misidentify which group actually performs best. With unequal success rates (best at p = 0.9, worst at 0.75, moderate at 0.8 and 0.85), GPT-4o and Gemini-2.5-Flash identified the best-performing group only 26.3% of the time in 40-round games, versus 29.0% for the second-best, 21.5% for the third-best, and 23.3% for the worst. Doubling to 80 rounds while informing the model of this produced no statistically significant improvement (26.2%, 30.5%, 24.4%, 18.9%).

• Additional features reduce stratification only when salient. In the resettlement task, adding age and education reduced SI by 0.43, 0.59, and 0.70 (single, other, both), while the less salient hair color and tattoo shape reduced it by only 0.25, 0.44, and 0.42. Gemini and Claude still retained SIs around 0.7 even with the informative features.

• Explicit incentivization of diversity is the single most effective intervention. Of four prompt steers (direct fairness instruction, internal values, societal norms, and an explicit diversity term in the objective), only the objective-level diversity steer robustly reduced stratification across direct and CoT prompting, producing SIs lower than both the random baseline and humans for nearly all models and prompts except Gemini. Claude 4 Sonnet refused to respond after the internal value steering prompt.

Methodology in Plain English

The researchers adapted an iterative hiring game from prior psychology work. A model is told it is a hiring consultant in a fictional city whose residents come from four ancestral villages: Tufa, Aima, Reku, and Weki. Over 40 rounds, each round presents a new job vacancy and one applicant from each group. The model picks a candidate, learns immediately whether the hire succeeded, and moves on. What the model is not told is that every group is equally likely to succeed at every job, with success drawn from Bernoulli(0.9). The jobs themselves span the two dimensions of social cognition — warmth and competence — yielding categories such as doctors (high competence, high warmth), lawyers (high competence, low warmth), childcare aides (low competence, high warmth), and janitors (low competence, low warmth).

For each model and prompt type, the authors ran n = 30 independent 40-round games with the job order shuffled each time. Models were prompted both directly and with chain-of-thought at default temperature, using medium reasoning effort for reasoning models. They computed three metrics over the runs: SI (how concentrated each group's allocations are relative to a uniform spread), BGD (how different the groups' allocation patterns are from one another), and GASI (how consistent a group's allocation is across independent runs). Statistical tests used a Benjamini-Hochberg correction with α = .05. Comparisons included human participants from the original study, fair random assignment, and Thompson sampling with a uniform prior.

The intervention studies varied model inputs (temperature at 1.5, CoT prompting, compressed context), the structure of the environment (lower success rates, LLM-elicited job-specific success rates, unequal group success rates, two new grounded settings — refugee resettlement and military conscription — an abstract setting with actions "A"–"D" and contexts "C1"–"C4", and added features such as age, education, hair color, and tattoo shape), and four prompt-steering conditions targeting fairness instructions, internal values, societal norms, and an explicit diversity term in the objective. Code is available at https://github.com/addisonwu05/LLM-Natural-Segregation.

Why This Matters

Impact on research. The paper reframes bias in LLMs as something that can be created rather than only reflected, distinguishing "allocational bias" (unequal distribution of outcomes) from the representational bias that dominates the fairness literature. It challenges the assumption that better benchmark scores on single-turn bias evaluations imply safer behavior over time, and it connects LLM behavior to classic work on statistical discrimination, in which unequal treatment can emerge from rational decision-making without any underlying taste for discrimination.

Real-world applications:

• Hiring and recruitment systems that screen or prioritize candidates across repeated decisions could entrench spurious group–role associations even when no group differences exist.
• Agentic assistants with persistent memory that allocate resources, schedule tasks, or assign personnel across many turns may drift toward stratification with no human oversight.
• Public-sector allocation tools, such as refugee resettlement or conscript assignment, both of which were directly tested here and both of which produced stratification (average SIs of 1.13 and 1.26).
• Content recommendation and engagement-optimizing platforms, which the authors use as an analogy: closed-loop optimization against a narrow objective produced societal harms such as echo chambers and polarization.

Industry relevance. The finding that newer, larger, and reasoning-capable models stratify more is directly relevant to deployment decisions, since capability upgrades are typically treated as unambiguous improvements. The paper's recommendation — that stronger optimizers need better-formulated objectives — implies that teams shipping agentic products should specify multifaceted goals that include societal values, not just task success.

Future Directions

• Designing intrinsic rather than prescriptive solutions. The authors note that the most effective intervention found here is an explicit prompt-level diversity incentive, but caution that blindly applying such steers can reduce overall success when underlying group success rates genuinely differ (Appendix K). They call for solutions that are intrinsic to the model rather than externally imposed.
• Evaluating long-term, downstream societal outcomes. The divergence between strong single-turn benchmark performance and severe emergent stratification suggests current evaluations are too isolated to capture how these systems shape outcomes over time in continuous deployment.
• Resolving the generalization tension in alignment. The same capacity for extrapolating patterns from few examples that enables few-shot learning also drives premature stratification, raising the open question of how to selectively discourage harmful pattern-matching without suppressing reasoning.
• Extending the paradigm to naturalistic difficulty. The authors observe that the settings producing ideal, non-stratified behavior (low success rates, aggregated history) are artificial, and ask how realistic levels of difficulty and noise would structure model allocations in practice.

Target Audience

This paper is most valuable to AI safety and fairness researchers, LLM evaluation and red-teaming practitioners, and social scientists studying discrimination and stereotype formation, particularly those interested in how algorithmic decision-making produces inequality through feedback loops. It is also relevant to product and policy teams deploying LLM agents with persistent memory or multi-turn autonomy, and to readers interested in the parallel between closed-loop optimization in AI and its consequences in social media and other large-scale systems. The interdisciplinary framing, combining psychology paradigms with machine learning evaluation, makes it accessible to readers with background in either field but assumes comfort with quantitative metrics and statistical reporting.

Authors’ abstract

As large language models (LLMs) are adopted into frameworks that grant them the capacity to make real decisions, it is increasingly important to ensure that they are unbiased. In this paper, we argue that the predominant approach of simply removing existing biases from models is not enough. Using a paradigm from the psychology literature, we demonstrate that LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist. These biases result in highly stratified task allocations, which are less fair than assignments by human participants and are exacerbated in newer and larger models. In humans, emergent biases like these have been shown to result from exploration-exploitation trade-offs, where the decision-maker explores too little, allowing early observations to strongly influence impressions about entire demographic groups. To alleviate this effect, we explore a series of interventions targeting model inputs, problem structure, and explicit steering. While most interventions have limited effect, explicitly incentivizing exploration robustly reduces stratification, highlighting the need for better multifaceted objectives to mitigate bias. These results reveal that LLMs are not merely passive mirrors of human social biases, but can actively create new ones from experience, raising urgent questions about how these systems will shape societies over time.